Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- 2 TB Sort
- #GEN
- yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar teragen -Dmapreduce.job.maps=76 20000000000 /tmp/gen-2tb/
- #SORT
- nohup yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar terasort -Dmapreduce.input.fileinputformat.split.minsize=500000000 -Dmapreduce.input.fileinputformat.split.maxsize=750000000 -Dmapreduce.map.sort.spill.percent=0.98 -Dmapreduce.job.reduces=38 -Dmapreduce.job.reduce.slowstart.completedmaps=0.4 -Dmapreduce.map.output.compress=true -Dmapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec /tmp/gen-2tb/ /tmp/sort-2tb/ > 2t.log &
Add Comment
Please, Sign In to add comment
