Pig Performance Measures

Performance tuning for pig script

       1. Avoid splits use filter

            Filter is a type of implicit split which splits the input according to the condition provided in the filter condition.

Example : 

Input 1

=======

1 2 3 4

5 6 7 8

0 46 47 48

Example1:

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int,c:int,d:int);

X = SPLIT A IF a >0 , B IF a>40;

DUMP X;

Example 2:

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int,c:int,d:int);

X = FILTER A BY a > 0;

Y = FILTER A BY a > 40;

DUMP X;

DUMP Y;

DESCRIPTION :

In above example using filter instead of split improves the performance of the pig.

This reduces processing a multiple times in same statement and also helps in intermediate

Storage of output which can be used for other processing.

2. Store Intermediate Results:

It is good to store intermediate results so that further processing’s can be done on top of that instead of processing the initial input again which is time consuming.

Example:

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int,c:int,d:int);

X = FILTER A BY a > 0;

L = GROUP X BY a;

Y = FOREACH L GENERATE SUM(b);

Z = FOREACH L GENERATE DIFFERENCE(b);

D= STORE Y IN ‘/home/hadoop/’;

In above example instead of reprocessing a to generate sum and difference of field it is better to store the result of L and do further processing on it. Which saves reprocessing of a again for same condition.

3. Remove Unnecessary dumps :

             DUMP is a interactive statement using which will disable multiquery processing and consumes more time than STORE statement.

Example:

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int,c:int,d:int);

X = FILTER A BY a > 0;

DUMP X;

L = GROUP X BY a;

Y = FOREACH L GENERATE SUM(b);

Z = FOREACH L GENERATE DIFFERENCE(b);

D= STORE Y IN ‘/home/hadoop/’;

Description :

        In above example aliases A and X will be processed which is dumped (which is treated as one job) and then A,X,L,Y,Z and D is performed so the same script is executed as two separate jobs because of dump statement. So remove all the pig scripts that are used for debugging.

4. Avoid using User Defined Functions:

           Unless it is necessary avoid UDF’s as it needs extra processing  other than running the pig scripts Which consumes time and CPU.Pig provides many built in functionalities which can be used in place of UDF’s

Some built in functions like SUM, COUNT,DISTINCT,FLATTEN,GROUPBY,FILTER can be used instead of elaborating everything in a UDF which is again wastage of time and memory.

5. Other ways to improve pig performance

a. Proper Usage of Operators:

 A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int,c:int,d:int);

 B= FILTER A BY a by NOT (NOT(a>5) OR a>10);

Can be simplified using a AND operator which saves extra burden of doing two negations and one or to obtain the result.

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int,c:int,d:int);

 B= FILTER A BY a a>5 AND a<10;

 b. Avoid unnecessary calculations

 A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int,c:int,d:int);

 B= FILTER A BY a > 1+2;

   Here instead of doing 1+2 operation for each value of a it can be simplified with pre computed value of 3.

B= FILTER A BY a > 3;

c. Avoid repetition of conditions in various forms

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int,c:int,d:int);

 B= FILTER A BY a > 0 and a > 3;

   Here if a is greater than 3 then obviously it should be greater than 0 hence this check can be avoided and simplified to

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int,c:int,d:int);

 B= FILTER A BY  a > 3;

4. Elimination of the condition which always results in result TRUE like

  B = FILTER A BY  1==1;

6. Minimize number of opertions performed

Example :

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int,c:int,d:int);

B = GROUP A BY a;

C = FOREACH B GENERATE a+b AS d,a-c AS e;

D = FOREACH C GENERATE d-c;

E = STORE D IN ‘/home/hadoop/output/’

Here above operation can be simplified to

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int,c:int,d:int);

B = GROUP A BY a;

C = FOREACH B GENERATE a+b-c AS d,a-c AS e;

E = STORE C IN ‘/home/hadoop/output/’

Since calculating a+b and then storing the result in a alias and subtracting c from it is same as doing a-b-c which saves time and memory.

8. Limit the number of output lines

          Store or dump only the number of lines you are interested by setting the limit.

         

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int,c:int,d:int);

B = FILTER A > 50;

C = LIMIT B 10;

Here instead of listing all the values of B , to view just 10 values limit it with the count.

PUSHUPFilters:

Instead of processing the entire data filter only the input that is needed and do processing on top of it .

Example :

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int,c:int,d:int);

B = FILTER A > 0;

C = FOREACH B GENERATE SUM(b);

In above example instead of computing sum for all the values of field b just filter the values that are greater than zero and perform the sum which prevents processing of  unwanted data.

9. Do Explicit Casting:

           Instead of allowing the pig to do the casting perform the casting explicitly when the data type is known which saves time.

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int,c:int,d:int);

As mentioned above load the data with explicit casting instead of

A = load '/home/hadoop/input/' using PigStorage('\t') as (a,b,c,d);

10  Filter only necessary fields while processing  :

      A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int,c:int,d:int,e:int);

      B = FILTER A BY a >0;

      C = FILTER B BY b > 50;

      B = FOR EACH GENERATE COUNT(c);

      D = FOR EACH B GENERATE COUNT(d);

      E = UNION B,D;

Here in above example field D is used nowhere in the processing so it can be removed after loading the data to alias A which will save time and memory and improve performance.

Above script can be modified as

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int,c:int,d:int,e:int);

A1 = FOREACH A generate a,b,c,d;

B = FILTER A1 BY a >0;

C = FILTER A1 BY b > 50;

      B = FOR EACH GENERATE COUNT(c);

      D = FOR EACH B GENERATE COUNT(d);

      E = UNION B,D;

11. UDF Considerations :

         UDF functions are used to add customized functionalities to pig script.UDF’s if not coded in proper way can degrade the performance of scripts.Following are the things that should be added while creating a UDF.

A.    Place the UDF classes in proper package

B.    Register the jars before using calling udf’s

C.    Don’t register unwanted jar in the pig script.

D.   Call the functions with proper package and class names.

12. Drop Null Values before calculations :

           Remove values that are having only values as null has null values no significance in join,sum,difference and other  algebric functions they are usually ignored.

For example pig can be written as

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int);

B = FILTER A BY a is not null;

C = FILTER A BY b is not null;

D = JOIN B by a,C by b;

13. Use the PARLLEL clause

          PARLLEL clause in a job lets us to set number of reducer jobs that can run in parllel. Default the PARLLEL value is set to 1.If this PARLLEL class is not set by default we will get only one reduce job which will affect the performance of the pig script.

Example you can set PARLLEL class in pig script as

SET DEFAULT_PARLLEL 10;

This will set number of reducer than can run parallel as 10.

14. Load only the needed input

          In a file if you want to do processing only in first 50 rows then just load only 50 rows instead of loading the entire file.

For Example

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int);

B = limit A 100;

15. DISTINCT VS GROUP BY

         If  you want to find distinct values of a field then there are two approaches to obtain it and they are GROUP BY and DISTINCT. It is always better to use DISTINCT instead of GROUP BY to boost the performance of pig script.

Example

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int);

B = FOREACH A GENERATE a;

C = GROUP B BY a;

Can be improved by changing it as

Example

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int);

B = FOREACH A GENERATE a;

C = distict b;

15. Filter Early an dofter

         If possible filter the values before doing a group by.Since combiner improves the performance of a script which can be be used only without operators between group by and for each.

Example:

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int);

B = FOREACH A GENERATE a;

C = GROUP B BY a;

D = FOREACH A GENERATE sum(b);

E = FILTER D BY b>0;

Can be optimized as

A = load '/home/hadoop/input/' using PigStorage('\t') as (a:int,b:int);

B = FOREACH A GENERATE a;

C = GROUP B BY a;

C1 = FILTER A BY b >0;

D = FOREACH A GENERATE sum(b);

16. Keep huge data last

While doing a join or cogroup with input of multiple size always keep the large data at the end

Doing so will improve the pig performance and will make it more efficient.