- Hadoop Real-World Solutions Cookbook(Second Edition)
- Tanmay Deshpande
- 498字
- 2021-07-09 20:02:52
Map Reduce program to find distinct values
In this recipe, we are going to learn how to write a map reduce program to find distinct values from a given set of data.
Getting ready
To perform this recipe, you should have a running Hadoop cluster as well as an eclipse that is similar to an IDE.
How to do it
Sometimes, there may be a chance that the data you have contains some duplicate values. In SQL, we have something called a distinct function, which helps us get distinct values. In this recipe, we are going to take a look at how we can get distinct values using map reduce programs.
Let's consider a use case where we have some user data with us, which contains two columns: userId
and username
. Let's assume that the data we have contains duplicate records, and for our processing needs, we only need distinct records through user IDs. Here is some sample data that we have where columns are separated by '|':
1|Tanmay 2|Ramesh 3|Ram 1|Tanmay 2|Ramesh 6|Rahul 6|Rahul 4|Sneha 4|Sneha
The idea here is to use the default reducer behavior where the same keys are sent to one reducer. In this case, we will make userId
the key and emit it to the reducer. In the reducer, the same keys will be reduced together, which will avoid duplicates.
Let's look at the Mapper Code.
public static class DistinctUserMapper extends Mapper<Object, Text, Text, NullWritable> { private Text userId = new Text(); public void map(Object key, Text value, Context context) throws IOException, InterruptedException { String words[] = value.toString().split("[|]"); userId.set(words[0]); context.write(userId, NullWritable.get()); } }
We only want distinct user IDs, hence, we emit only user IDs as keys and nulls as values.
Now, let's look at the reducer code:
public static class DistinctUserReducer extends Reducer<Text, NullWritable, Text, NullWritable> { public void reduce(Text key, Iterable<NullWritable> values, Context context) throws IOException, InterruptedException { context.write(key, NullWritable.get()); } }
Here, we only emit user IDs as they come. This step removes duplicates as the reducer only treats the records by their keys and only one record per key is kept.
The driver code remains simple, as shown here:
public class DistinctValues { public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); if (args.length != 2) { System.err.println("Usage: DistinctValues <in><out>"); System.exit(2); } Job job = Job.getInstance(conf, "Distinct User Id finder"); job.setJarByClass(DistinctValues.class); job.setMapperClass(DistinctUserMapper.class); job.setReducerClass(DistinctUserReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(NullWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } }
Now, when we execute the code, we will see the following output:
hadoop jar distinct.jar com.demo.DistinctValues /users /distinct_user_ids hadoop fs -cat /distinct_user_ids/part-r-00000 1 2 3 4 6
How it works...
When mapper emits keys and values, the output is shuffled across the nodes in the cluster. Here, the partitioner decides which keys should be reduced and on which node. On all the nodes, the same partitioning logic is used, which makes sure that the same keys are grouped together. In the preceding code, we use this default behavior to find distinct user IDs.
- AutoCAD繪圖實用速查通典
- Circos Data Visualization How-to
- ETL with Azure Cookbook
- Maya 2012從入門到精通
- Apache Spark Deep Learning Cookbook
- 網絡服務搭建、配置與管理大全(Linux版)
- Dreamweaver CS6中文版多功能教材
- Excel 2007終極技巧金典
- Spark大數據商業實戰三部曲:內核解密|商業案例|性能調優
- Mastering Predictive Analytics with scikit:learn and TensorFlow
- 深度學習原理與 TensorFlow實踐
- 基于Proteus的PIC單片機C語言程序設計與仿真
- 精通ROS機器人編程(原書第2版)
- 特征工程入門與實踐
- Flash CS3動畫制作融會貫通