The InputFormat class

The input data specification for a MapReduce Hadoop job is given via the InputFormat hierarchy of classes. The InputFormat class family has the following main functions:

The FileInputFormat subclass and associated classes are commonly used for jobs that take inputs from HFDS. The DBInputFormat subclass is a specialized class that can be used to read data from a SQL database. CombineFileInputFormat is the direct abstract subclass of the FileInputFormat class, which can combine multiple files into a single split.

The InputSplit class

The abstract InputSplit class and its associated concrete classes represent a byte view of the input data. An InputSplit class is characterized by the following main attributes:

In HDFS, an InputSplit class is created per file if the file size is less than the HDFS block size. For example, if the HDFS block size is 128 MB, any file with a size less than 128 MB resides in its own InputSplit class. For files that are broken up into blocks (size of the file is greater than the HDFS block size), a more complex formula is used to calculate InputSplit. The InputSplit class has an upper bound of the HDFS block size, unless the minimum size of a split is greater than the block size. Such cases are rare and could lead to locality problems.

Based on the locations of the splits and the availability of resources, the scheduler makes a decision on which node should execute the Map task for that split. The split is then communicated to the node that executes the task.

InputSplitSize = Maximum(minSplitSize, Minimum(blocksize, maxSplitSize)) 

minSplitSize : mapreduce.input.fileinputformat.split.minsize

blocksize: dfs.blocksize 
maxSplitSize - mapreduce.input.fileinputformat.split.maxsize