Advertisement
Guest User

Untitled

a guest
Mar 28th, 2017
54
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 2.10 KB | None | 0 0
  1. python my_script.py -r emr
  2. --emr-job-flow-id=j-JOBID --conf-path=./mrjob.conf --no-output
  3. --output hdfs:///my-output
  4. hdfs:///my-input-directory/my-files*.gz
  5.  
  6. Traceback (most recent call last):
  7. File "pipes/sampler.py", line 28, in <module>
  8. SamplerJob.run()
  9. File "/Library/Python/2.7/site-packages/mrjob/job.py", line 483, in run
  10. mr_job.execute()
  11. File "/Library/Python/2.7/site-packages/mrjob/job.py", line 501, in execute
  12. super(MRJob, self).execute()
  13. File "/Library/Python/2.7/site-packages/mrjob/launch.py", line 146, in execute
  14. self.run_job()
  15. File "/Library/Python/2.7/site-packages/mrjob/launch.py", line 206, in run_job
  16. with self.make_runner() as runner:
  17. File "/Library/Python/2.7/site-packages/mrjob/job.py", line 524, in make_runner
  18. return super(MRJob, self).make_runner()
  19. File "/Library/Python/2.7/site-packages/mrjob/launch.py", line 161, in make_runner
  20. return EMRJobRunner(**self.emr_job_runner_kwargs())
  21. File "/Library/Python/2.7/site-packages/mrjob/emr.py", line 585, in __init__
  22. self._output_dir = self._check_and_fix_s3_dir(self._output_dir)
  23. File "/Library/Python/2.7/site-packages/mrjob/emr.py", line 776, in _check_and_fix_s3_dir
  24. raise ValueError('Invalid S3 URI: %r' % s3_uri)
  25. ValueError: Invalid S3 URI: 'hdfs:///input/sample'
  26.  
  27. > export jobflow=$(elastic-mapreduce --create --alive --plain-output
  28. > --master-instance-type m1.small --slave-instance-type m1.xlarge --num-instances 21 --name "Custer Name" --bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-hadoop --args
  29. > "--mapred-config-file,s3://myBucket/conf/custom-mapred-config-file.xml")
  30. >
  31. >
  32. > elastic-mapreduce -j $jobflow --jar
  33. > s3://us-east-1.elasticmapreduce/libs/s3distcp/1.latest/s3distcp.jar
  34. > --arg --src --arg 's3://myBucket/input/' --arg --dest --arg 'hdfs:///input'
  35. >
  36. > elastic-mapreduce --jobflow $jobflow --jar s3://myBucket/bin/step1.jar
  37. > --arg hdfs:///input --arg hdfs:///output-step1 --step-name "Step 1"
  38. >
  39. > elastic-mapreduce --jobflow $jobflow --jar s3://myBucket/bin/step2.jar
  40. > --arg hdfs:///input,hdfs:///output-step1 --arg s3://myBucket/output/ --step-name "Step 2"
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement