Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- beat the Cisco UCS 15 Worker benchmark with our 13 Worker cluster (with alittle better hardware.)
- For 10TB we took about 87 minutes when the benchmark is 110-115 minutes. This was with 6.6GB Maps (1.7GB IO Buffer) and Reduces.
- -Dmapreduce.input.fileinputformat.split.minsize=1000000000
- -Dmapreduce.input.fileinputformat.split.maxsize=1500000000
- -Dmapreduce.map.sort.spill.percent=0.98
- ​Because we have a MapSide Buffer of 1.7GB we want to make the best use of it. So rather then having the default behavior of 1 Map to 1 Block (128mb) we are going to send 1-1.5GB worth of blocks to each Mapper. With spill set to 0.98% we will read the entire file into the buffer (1.5GB data < 1.7GB buffer) without pausing to spill and then read again to spill a final time, so this way we just spill once.
- -Dmapreduce.job.reduces=240
- Cisco's recommendation, this gives each reducer about 41gb of data to sort per reduce. You don't want to many because you can't make good use of both CPU and IO. Aim for around 15 min task completion time and check the counters to see if they spill alot.
- -Dmapreduce.job.reduce.slowstart.completedmaps=0.4
- Because reduces cannot complete until all maps have been completed we want to start the reduces only after 40% of maps are completed so they don't steal resources we could use for completing maps faster.
- -Dmapreduce.map.output.compress=true
- -Dmapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec
- Compression is King in most cases. They had plenty of spare CPU so by using mapside compression we reduce the amount of data which must be written to disk for the shuffle, which in turn reduces how much to be sent over the network.
Advertisement
Add Comment
Please, Sign In to add comment
