MapReduce

-

Introduction To MapReduce:

·       MapReduce is a computing model that decomposes larger manipulation jobs  into individual tasks.

·       These tasks can be executed parallel across the cluster.

·       The results of the tasks are joined together to form the final result.

·       MapReduce is the data processing component of Hadoop.

·       Mapreduce transforms the list of input data into list of output data elements.

·       Mapreduce is the heart of hadoop. It is designed for processing huge amount of data.

·       There are two different processing layers:

1.    Map

2.    Reduce

Different Phases in Mapreduce :

 

 

 

Map:

·       Map takes the set of data & convert into another set of data where individual elements are broken down into tuples(key, value pairs).

·       Here data can be in structured or unstructured format.

·       Key is reference to input value.(IntWritable, LongWritable)

·       Value is a dataset on which to operate.(IntWritable, LongWritable, TextWritable).

·       The output of map is  known as intermediate output, which is stored in local system.

·       The intermediate output of map is given as input to reduce phase.

·       If there is no reduce phase or if the processing of reduce is completed then,  the output is stored on hdfs.

·       The movement of output from mapper phase to reducer phase is known as shuffling.

·       The output of mapper can be different from input pair.

·       Different phases under map phase:

Ø Partitioner:Output of mapper is partitioned and filtered to many partitions by partitioner.

Ø Combiner: Before passing the output to reduce phase, combiner summarizes the output record with same key. Therefore combiner is known as “Mini-Reducer”

Reduce:

·       The input of reducer is intermediate output which is produced by mapper.

·       Keys , Value pairs provided to reducer are sortedby key.

·       Reducer is the second phase of map reduce.

·       An output of reduce phase is final output.

·       Different  aggregate operations like filter etc.,,  can be performed on reduce phase.

·       By default number of reducers is 1.

·       There are 3 phases of reducer in Mapreduce:

Ø Shuffling: The process of transferring output from mappers to reducers is known as shuffling.

Ø Sorting: The keys generated by mapper are automatically sorted by mapreduce. Values generated to the reducer are sorted which helps reducer, to easily distinguish when, a new reduce task should start.

Ø Reduce phase: Final output  is produced , after sorting and aggregate operations are performed.

·       We can set the count of reducers by using the method as follows:

Job.setNumReducerTask(Int).

By increasing the number of reducers,

Ø It increases the framework overhead

Ø Increases load balancing

Ø Lowers the cost of failures.

 

DataTypes:

Normal DataType                    MapReduce DataType

1.     Int                        :         Intwritable.

2.     Float                     :         Floatwritable.

3.     Double                 :         Double writable.

4.     Long                     :         Long Writable.

5.     String                   :         StringWritable.

6.     Boolean                :         BooleanWritable.

 


Comments

Post a Comment

Popular posts from this blog

Hadoop

-
Image
“90% of the world’s data was generated in the last few years.” Due to the advent of new technologies, devices, and communication means like social networking sites, the amount of data produced by mankind is growing rapidly every year. The amount of data produced by us from the beginning of time till 2003 was 5 billion gigabytes. If you pile up the data in the form of disks it may fill an entire football field. The same amount was created in every two days in  2011 , and in every ten minutes in  2013 . This rate is still growing enormously. Though all this information produced is meaningful and can be useful when processed, it is being neglected. What is Big Data? Big data  is a collection of large datasets that cannot be processed using traditional computing techniques. It is not a single technique or a tool, rather it has become a complete subject, which involves various tools, technqiues and frameworks. What Comes Under Big Data? Big data involves the data p...
Read more

Problem Statement Of Real Estate Use Cases

-
Hive Use Case – Real Estate Analysis   Due to an industry, real estate activity is outlined as any economic dealings associated with the acquisition, sale, owner-operation or lease of property. This in addition includes income-generating residential properties, like flat, buildings and single-room rentals. Real estate services are not enclosed within the sector. Also, samples of real estate services embrace brokerages, property management, appraisers, investment property analysts and different consultants. All analysts, working with Big Data are using Hive or some other tool to query dataset and get results with ease. Although other querying languages exists, Hive gives us a variety of new features when compared to traditional approaches. So,On demand of accelerating consumers in real estate field, filtered series of information was collected and handed over to data analyst team. Load data into Hive. Problem Statement:        ...
Read more

Problem Statement Of Bank Marketing analysis

-
Analyze Marketing data for call campaign by bank Try yourself! Your client- a Portuguese banking institution—ran a marketing campaign to convince potential customers to invest in bank term deposit. The data is related to direct marketing campaigns of the bank. The marketing campaigns were based on phone calls. Often, more than one contact by phone to the same client was required, in order to assess if the product (bank term deposit) would be ('yes') or not ('no') subscribed: Data Fields age;"job";"marital";"education";"default";"balance";"housing";"loan";"contact";"day";"month";"duration";"campaign";"pdays";"previous";"poutcome";"y" 1 - age (numeric) 2 - job : type of job (categorical: 'admin.','blue-collar','entrepreneur','housemaid','management','retired...
Read more