官术网_书友最值得收藏!

Block size and number of mappers and reducers

An important consideration in the MapReduce process is an understanding of HDFS block size, that is, the size of the chunks into which the files have been split. A MapReduce task that needs to access a certain file will need to perform the map operation on each block representing the file. For example, given a 512 MB file and a 128 MB block size, four blocks would be needed to store the entire file. Hence, a MapReduce operation will at a minimum require four map tasks whereby each map operation would be applied to each subset of the data (that is, each of the four blocks).

If the file was very large, however, and required say, 10,000 blocks to store, this means we would have required 10,000 map operations. But, if we had only 10 servers, then we'd have to send 1,000 map operations to each server. This might be sub-optimal as it can lead to a high penalty due to disk I/O operations and resource allocation settings on a per-map basis.

The number of reducers required is summarized very elegantly on Hadoop Wiki (https://wiki.apache.org/hadoop/HowManyMapsAndReduces).

The ideal reducers should be the optimal value that gets them closest to:
* A multiple of the block size * A task time between 5 and 15 minutes * Creates the fewest files possible
Anything other than that means there is a good chance your reducers are less than great. There is a tremendous tendency for users to use a REALLY high value ("More parallelism means faster!") or a REALLY low value ("I don't want to blow my namespace quota!"). Both are equally dangerous, resulting in one or more of:
* Terrible performance on the next phase of the workflow * Terrible performance due to the shuffle * Terrible overall performance because you've overloaded the namenode with objects that are ultimately useless * Destroying disk IO for no really sane reason * Lots of network transfers due to dealing with crazy amounts of CFIF/MFIF work

主站蜘蛛池模板: 大厂| 滨海县| 会理县| 沂水县| 乳源| 桐梓县| 神农架林区| 蒙阴县| 工布江达县| 聂拉木县| 讷河市| 荆门市| 报价| 海门市| 湖北省| 壤塘县| 乌恰县| 车致| 海安县| 淮北市| 凤阳县| 萝北县| 南汇区| 治县。| 当涂县| 柳林县| 万载县| 依兰县| 卫辉市| 嘉荫县| 青河县| 营山县| 岳阳县| 新巴尔虎右旗| 渝中区| 北碚区| 平遥县| 临桂县| 吉安市| 哈密市| 谷城县|