- Mastering Hadoop
- Sandeep Karanth
- 263字
- 2021-08-06 19:53:01
Chapter 3. Advanced Pig
Running Java MapReduce jobs on Hadoop provides the most flexibility with the least abstraction. However, abstractions are necessary to infer patterns, accomplish common data manipulation tasks, reduce complexity, and flatten the learning curve. Pig is a platform that provides a framework and high-level abstractions to build MapReduce programs for Hadoop. It has a scripting language called Pig Latin. Pig Latin can be compared to SQL in terms of operator capabilities.
Developed at Yahoo! around the year 2006, Pig was used as a framework to specify ad hoc MapReduce workflows. In the following year, it was moved to Apache Software Foundation. The latest release of Pig is 0.12.1.
Tip
The official release of Pig is currently incompatible with Hadoop 2.2.0. It expects libraries from Hadoop 1.2.1. Running any Pig script fails, with the following exception:
Unexpected System Error Occured: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected.
Fixing this requires a recompile of the Pig binaries. Run the following command and replace the newly generated pig.jar and pig-withouthadoop.jar files:
ant clean jar-all -Dhadoopversion=23
In this chapter, we will look at the advanced features of Pig by:
- Looking at how Pig is different when compared to SQL
- Analyzing how Pig Latin scripts are translated to MapReduce programs
- Delving into the advanced relational operators that Pig supports; we will delve deep into these relational operators and look at their applications with examples
- Studying ways to extend Pig beyond its off-the-shelf capabilities using User-defined Functions or UDFs that can implement a variety of interfaces; we will examine some of these interfaces