Pig 1

-


APACHE PIG

Pig Release Date: 2008

At Yahoo! 40% of all Hadoop jobs are run with Pig. Come join us!
Apache Pig – History
In 2006, Apache Pig was developed as a research project at Yahoo, especially to create and execute MapReduce jobs on every dataset.
In 2007, Apache Pig was open sourced via Apache incubator. In 2008,
the first release of Apache Pig came out. In 2010,
Apache Pig graduated as an Apache top-level project.
The language used to analyze data in Hadoop using Pig is known as Pig Latin.
 It is a highlevel data processing language which provides a rich set of data types and operators to perform various operations on the data.
To analyze data using Apache Pig, programmers need to write scripts using Pig Latin language. All these scripts are internally converted to Map and Reduce tasks.
To perform a particular task Programmers using Pig, programmers need to write a Pig script using the Pig Latin language, and execute them using any of the execution mechanisms (Grunt Shell, UDFs, Embedded). After execution, these scripts will go through a series of transformations applied by the Pig Framework, to produce the desired output.
Internally, Apache Pig converts these scripts into a series of MapReduce jobs, and thus, it makes the programmer’s job easy. The architecture of Apache Pig is shown below

What is Apache Pig?
Pig is a tool/platform which is used to analyze larger sets of data representing them as data flows. Pig is generally used with Hadoop; we can perform all the data manipulation operations in Hadoop using Apache Pig.
To write data analysis programs, Pig provides a high-level language known as Pig Latin. This language provides various operators using which programmers can develop their own functions for reading, writing, and processing data.
Apache Pig has a component known as Pig Engine that accepts the Pig Latin scripts as input and converts those scripts into MapReduce jobs.
Why name PIG?
During development the YAHOO engineers name the language as "THE LANGUAGE" and after that the renamed it "PIG".
Pig properties?
pig can eat anything.
pig cant fly.
WHAT PIG DOES?
Pig was designed for performing a long series of data operations, making it ideal for three categories of Big Data jobs:
·         Extract-transform-load (ETL) data pipelines,
·         Research on raw data, and
·         Iterative data processing.

HOW PIG WORKS?

·         MapReduce Mode. This is the default mode, which requires access to a Hadoop cluster.
·         Local Mode. With access to a single machine, all files are installed and run using a local host and file system.

Why Do We Need Apache Pig?

Programmers who are not so good at Java normally used to struggle working with Hadoop,
especially while performing any MapReduce tasks.
 Apache Pig is a boon for all such programmers.
·        Using Pig Latin, programmers can perform MapReduce tasks easily without having to type complex codes in Java.
·        Apache Pig uses multi-query approach, thereby reducing the length of codes. Pig Latin is SQL-like language and it is easy to learn Apache Pig when you are familiar with SQL.
·        Apache Pig provides many built-in operators to support data operations like joins, filters, ordering, etc. In addition, it also provides nested data types like tuples, bags, and maps that are missing from MapReduce.
  • Define a relation with and without schema
  • Define a new relation from an existing relation
  • Select specific columns from within a relation
  • Join two relations
  • Sort the data using ‘ORDER BY’
  • FILTER and Group the data using ‘GROUP BY’

Features of Pig

Apache Pig comes with the following features −
·        Rich set of operators − It provides many operators to perform operations like join, sort, filer, etc.
·        Ease of programming − Pig Latin is similar to SQL and it is easy to write a Pig script if you are good at SQL.
·        Optimization opportunities − The tasks in Apache Pig optimize their execution automatically, so the programmers need to focus only on semantics of the language.
·        Extensibility − Using the existing operators, users can develop their own functions to read, process, and write data.
·        UDF’s − Pig provides the facility to create User-defined Functions in other programming languages such as Java and invoke or embed them in Pig Scripts.
·        Handles all kinds of data − Apache Pig analyzes all kinds of data, both structured as well as unstructured. It stores the results in HDFS.
·                   Apache Pig Vs MapReduce
·         Listed below are the major differences between Apache Pig and MapReduce.

Apache Pig
MapReduce
Apache Pig is a data flow language.
MapReduce is a data processing paradigm.
It is a high level language.
MapReduce is low level and rigid.
Performing a Join operation in Apache Pig is pretty simple.
It is quite difficult in MapReduce to perform a Join operation between datasets.
Any novice programmer with a basic knowledge of SQL can work conveniently with Apache Pig.
Exposure to Java is must to work with MapReduce.
Apache Pig uses multi-query approach, thereby reducing the length of the codes to a great extent.
MapReduce will require almost 20 times more the number of lines to perform the same task.
There is no need for compilation. On execution, every Apache Pig operator is converted internally into a MapReduce job.
MapReduce jobs have a long compilation process.
·                    
·                   Apache Pig Vs SQL
·         Listed below are the major differences between Apache Pig and SQL.
Pig
SQL
Pig Latin is a procedural language.
SQL is a declarative language.
In Apache Pig, schema is optional. We can store data without designing a schema (values are stored as $01, $02 etc.)
Schema is mandatory in SQL.
The data model in Apache Pig is nested relational.
The data model used in SQL is flat relational.
Apache Pig provides limited opportunity for Query optimization.
There is more opportunity for query optimization in SQL.



Pig Latin – Data types

Given below table describes the Pig Latin data types.
S.N.
Data Type
Description & Example
1
int
Represents a signed 32-bit integer.
Example : 8
2
long
Represents a signed 64-bit integer.
Example : 5L
3
float
Represents a signed 32-bit floating point.
Example : 5.5F
4
double
Represents a 64-bit floating point.
Example : 10.5
5
chararray
Represents a character array (string) in Unicode UTF-8 format.
Example : ‘tutorials point’
6
Bytearray
Represents a Byte array (blob).
7
Boolean
Represents a Boolean value.
Example : true/ false.
8
Datetime
Represents a date-time.
Example : 1970-01-01T00:00:00.000+00:00
9
Biginteger
Represents a Java BigInteger.
Example : 60708090709
10
Bigdecimal
Represents a Java BigDecimal
Example : 185.98376256272893883
Complex Types
11
Tuple
A tuple is an ordered set of fields.
Example : (raja, 30)
12
Bag
A bag is a collection of tuples.
Example : {(raju,30),(Mohhammad,45)}
13
Map
A Map is a set of key-value pairs.
Example : [ ‘name’#’Raju’, ‘age’#30]
 

Student_data = LOAD 'student_data.txt' USING PigStorage(',')as 
   ( id:int, firstname:chararray, lastname:chararray, phone:chararray, city:chararray );



Comments

Post a Comment

Popular posts from this blog

Problem Statement Of Real Estate Use Cases

-
Hive Use Case – Real Estate Analysis   Due to an industry, real estate activity is outlined as any economic dealings associated with the acquisition, sale, owner-operation or lease of property. This in addition includes income-generating residential properties, like flat, buildings and single-room rentals. Real estate services are not enclosed within the sector. Also, samples of real estate services embrace brokerages, property management, appraisers, investment property analysts and different consultants. All analysts, working with Big Data are using Hive or some other tool to query dataset and get results with ease. Although other querying languages exists, Hive gives us a variety of new features when compared to traditional approaches. So,On demand of accelerating consumers in real estate field, filtered series of information was collected and handed over to data analyst team. Load data into Hive. Problem Statement:        ...
Read more

Problem Statement Of Bank Marketing analysis

-
Analyze Marketing data for call campaign by bank Try yourself! Your client- a Portuguese banking institution—ran a marketing campaign to convince potential customers to invest in bank term deposit. The data is related to direct marketing campaigns of the bank. The marketing campaigns were based on phone calls. Often, more than one contact by phone to the same client was required, in order to assess if the product (bank term deposit) would be ('yes') or not ('no') subscribed: Data Fields age;"job";"marital";"education";"default";"balance";"housing";"loan";"contact";"day";"month";"duration";"campaign";"pdays";"previous";"poutcome";"y" 1 - age (numeric) 2 - job : type of job (categorical: 'admin.','blue-collar','entrepreneur','housemaid','management','retired...
Read more

Hadoop Comand

-
Syntax     :   hadoop fsck / COMMAND_OPTION Description path Start checking from this path. -delete Delete corrupted files. -files Print out files being checked. -files   -blocks Print out the block report -files   -blocks   -locations Print out locations for every block. -files   -blocks   -racks Print out network topology for data-node locations. -includeSnapshots Include snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it. -list-corruptfileblocks Print out list of missing blocks and files they belong to. -move Move corrupted files to /lost+found. -openforwrite Print out files opened for write.   HDFS support...
Read more