SHARE
TWEET

Untitled

a guest Jul 21st, 2019 63 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. # HDFS Write
  2. hadoop jar hadoop-mapreduce-client-jobclient-3.0.0-cdh6.2.0-tests.jar TestDFSIO \
  3. -Dmapred.job.queue.name=q2 \
  4. -write -nrFiles 50 -size 100MB
  5.  
  6. # HDFS Read
  7. hadoop jar hadoop-mapreduce-client-jobclient-3.0.0-cdh6.2.0-tests.jar TestDFSIO \
  8. -Dmapred.job.queue.name=q2 \
  9. -read -nrFiles 50 -size 100MB
  10.  
  11. # S3 Write
  12. hadoop jar hadoop-mapreduce-client-jobclient-3.0.0-cdh6.2.0-tests.jar TestDFSIO \
  13. -Dtest.build.data=s3://bucket/benchmarks/TestDFSIO \
  14. -Dfs.defaultFS=s3://bucket \
  15. -Dyarn.app.mapreduce.am.staging-dir=s3://bucket/tmp3 \
  16. -Dmapred.job.queue.name=q2 \
  17. -write -nrFiles 50 -size 100MB
  18.  
  19. # S3 Read
  20. hadoop jar hadoop-mapreduce-client-jobclient-3.0.0-cdh6.2.0-tests.jar TestDFSIO \
  21. -Dtest.build.data=s3://bucket/benchmarks/TestDFSIO \
  22. -Dfs.defaultFS=s3://bucket \
  23. -Dyarn.app.mapreduce.am.staging-dir=s3://bucket/tmp4 \
  24. -Dmapred.job.queue.name=q2 \
  25. -read -nrFiles 50 -size 100MB
  26.  
  27. # 처리 결과
  28. 19/06/18 01:32:50 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
  29. 19/06/18 01:32:50 INFO fs.TestDFSIO:            Date & time: Tue Jun 18 01:32:50 UTC 2019  
  30. 19/06/18 01:32:50 INFO fs.TestDFSIO:        Number of files: 5                             
  31. 19/06/18 01:32:50 INFO fs.TestDFSIO: Total MBytes processed: 500.0                         
  32. 19/06/18 01:32:50 INFO fs.TestDFSIO:      Throughput mb/sec: 91.35757354284671
  33. 19/06/18 01:32:50 INFO fs.TestDFSIO: Average IO rate mb/sec: 91.58104705810547             
  34. 19/06/18 01:32:50 INFO fs.TestDFSIO:  IO rate std deviation: 4.448786638351144
  35. 19/06/18 01:32:50 INFO fs.TestDFSIO:     Test exec time sec: 21.936
  36. 19/06/18 01:32:50 INFO fs.TestDFSIO:
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top