API tools faq
paste
Guest User

Untitled

a guest
Nov 20th, 2018
109
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 15.96 KB | None | 0 0
raw download clone embed print report
  1. bash-3.2$ pig -version
  2. Warning: $HADOOP_HOME is deprecated.
  3.  
  4. Apache Pig version 0.10.0 (r1328203)
  5. compiled Apr 19 2012, 22:54:12
  6. bash-3.2$ pig -p docPath=./data/rain.txt -p wcPath=./output/wc -p stopPath=./data/en.stop ./src/scripts/wc.pig
  7. Warning: $HADOOP_HOME is deprecated.
  8.  
  9. 2012-08-28 10:01:47,966 [main] INFO org.apache.pig.Main - Apache Pig version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
  10. 2012-08-28 10:01:47,967 [main] INFO org.apache.pig.Main - Logging error messages to: /Users/ceteri/src/concur/Impatient/part4/pig_1346173307964.log
  11. 2012-08-28 10:01:48.060 java[73829:1903] Unable to load realm info from SCDynamicStore
  12. 2012-08-28 10:01:48,252 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
  13. 2012-08-28 10:01:48,820 [main] WARN org.apache.pig.PigServer - Encountered Warning USING_OVERLOADED_FUNCTION 1 time(s).
  14. 2012-08-28 10:01:48,820 [main] WARN org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_CHARARRAY 2 time(s).
  15. 2012-08-28 10:01:48,968 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
  16. 2012-08-28 10:01:48,977 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
  17. 2012-08-28 10:01:48,994 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 2
  18. 2012-08-28 10:01:48,994 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 2
  19. 2012-08-28 10:01:49,030 [main] WARN org.apache.pig.PigServer - Encountered Warning USING_OVERLOADED_FUNCTION 1 time(s).
  20. 2012-08-28 10:01:49,030 [main] WARN org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_CHARARRAY 2 time(s).
  21. 2012-08-28 10:01:49,032 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: HASH_JOIN,GROUP_BY,FILTER
  22. 2012-08-28 10:01:49,064 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
  23. 2012-08-28 10:01:49,068 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
  24. 2012-08-28 10:01:49,070 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 2
  25. 2012-08-28 10:01:49,070 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 2
  26. 2012-08-28 10:01:49,086 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
  27. 2012-08-28 10:01:49,095 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
  28. 2012-08-28 10:01:49,098 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job7509048591395546006.jar
  29. 2012-08-28 10:01:53,505 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job7509048591395546006.jar created
  30. 2012-08-28 10:01:53,516 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
  31. 2012-08-28 10:01:53,524 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=1054
  32. 2012-08-28 10:01:53,524 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
  33. 2012-08-28 10:01:53,603 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
  34. 2012-08-28 10:01:53,615 [Thread-6] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  35. 2012-08-28 10:01:53,754 [Thread-6] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
  36. 2012-08-28 10:01:53,754 [Thread-6] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
  37. 2012-08-28 10:01:53,763 [Thread-6] WARN org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library not loaded
  38. 2012-08-28 10:01:53,766 [Thread-6] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
  39. 2012-08-28 10:01:53,775 [Thread-6] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
  40. 2012-08-28 10:01:53,775 [Thread-6] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
  41. 2012-08-28 10:01:53,776 [Thread-6] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
  42. 2012-08-28 10:01:54,021 [Thread-7] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : null
  43. 2012-08-28 10:01:54,037 [Thread-7] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed file:/Users/ceteri/src/concur/Impatient/part4/data/en.stop:0+544
  44. 2012-08-28 10:01:54,045 [Thread-7] INFO org.apache.hadoop.mapred.MapTask - io.sort.mb = 100
  45. 2012-08-28 10:01:54,183 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001
  46. 2012-08-28 10:01:54,184 [Thread-7] INFO org.apache.hadoop.mapred.MapTask - data buffer = 79691776/99614720
  47. 2012-08-28 10:01:54,184 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
  48. 2012-08-28 10:01:54,185 [Thread-7] INFO org.apache.hadoop.mapred.MapTask - record buffer = 262144/327680
  49. 2012-08-28 10:01:54,233 [Thread-7] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Created input record counter: Input records from _1_en.stop
  50. 2012-08-28 10:01:54,251 [Thread-7] INFO org.apache.hadoop.mapred.MapTask - Starting flush of map output
  51. 2012-08-28 10:01:54,262 [Thread-7] INFO org.apache.hadoop.mapred.MapTask - Finished spill 0
  52. 2012-08-28 10:01:54,265 [Thread-7] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
  53. 2012-08-28 10:01:57,005 [Thread-7] INFO org.apache.hadoop.mapred.LocalJobRunner -
  54. 2012-08-28 10:01:57,005 [Thread-7] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0001_m_000000_0' done.
  55. 2012-08-28 10:01:57,008 [Thread-7] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : null
  56. 2012-08-28 10:01:57,012 [Thread-7] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed file:/Users/ceteri/src/concur/Impatient/part4/data/rain.txt:0+510
  57. 2012-08-28 10:01:57,013 [Thread-7] INFO org.apache.hadoop.mapred.MapTask - io.sort.mb = 100
  58. 2012-08-28 10:01:57,073 [Thread-7] INFO org.apache.hadoop.mapred.MapTask - data buffer = 79691776/99614720
  59. 2012-08-28 10:01:57,073 [Thread-7] INFO org.apache.hadoop.mapred.MapTask - record buffer = 262144/327680
  60. 2012-08-28 10:01:57,087 [Thread-7] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Created input record counter: Input records from _0_rain.txt
  61. 2012-08-28 10:01:57,096 [Thread-7] INFO org.apache.hadoop.mapred.MapTask - Starting flush of map output
  62. 2012-08-28 10:01:57,098 [Thread-7] INFO org.apache.hadoop.mapred.MapTask - Finished spill 0
  63. 2012-08-28 10:01:57,100 [Thread-7] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting
  64. 2012-08-28 10:02:00,006 [Thread-7] INFO org.apache.hadoop.mapred.LocalJobRunner -
  65. 2012-08-28 10:02:00,007 [Thread-7] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0001_m_000001_0' done.
  66. 2012-08-28 10:02:00,018 [Thread-7] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : null
  67. 2012-08-28 10:02:00,018 [Thread-7] INFO org.apache.hadoop.mapred.LocalJobRunner -
  68. 2012-08-28 10:02:00,021 [Thread-7] INFO org.apache.hadoop.mapred.Merger - Merging 2 sorted segments
  69. 2012-08-28 10:02:00,028 [Thread-7] INFO org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 2 segments left of total size: 3284 bytes
  70. 2012-08-28 10:02:00,029 [Thread-7] INFO org.apache.hadoop.mapred.LocalJobRunner -
  71. 2012-08-28 10:02:00,061 [Thread-7] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
  72. 2012-08-28 10:02:00,062 [Thread-7] INFO org.apache.hadoop.mapred.LocalJobRunner -
  73. 2012-08-28 10:02:00,062 [Thread-7] INFO org.apache.hadoop.mapred.Task - Task attempt_local_0001_r_000000_0 is allowed to commit now
  74. 2012-08-28 10:02:00,064 [Thread-7] INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local_0001_r_000000_0' to file:/tmp/temp382990916/tmp-2005021410
  75. 2012-08-28 10:02:03,019 [Thread-7] INFO org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce
  76. 2012-08-28 10:02:03,020 [Thread-7] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0001_r_000000_0' done.
  77. 2012-08-28 10:02:03,022 [Thread-7] WARN org.apache.hadoop.mapred.FileOutputCommitter - Output path is null in cleanup
  78. 2012-08-28 10:02:04,203 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
  79. 2012-08-28 10:02:04,205 [main] WARN org.apache.pig.tools.pigstats.PigStatsUtil - Failed to get RunningJob for job job_local_0001
  80. 2012-08-28 10:02:04,207 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
  81. 2012-08-28 10:02:04,207 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
  82. 2012-08-28 10:02:04,208 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job6242959626572154070.jar
  83. 2012-08-28 10:02:07,871 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job6242959626572154070.jar created
  84. 2012-08-28 10:02:07,875 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
  85. 2012-08-28 10:02:07,879 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=1037
  86. 2012-08-28 10:02:07,879 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
  87. 2012-08-28 10:02:07,890 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
  88. 2012-08-28 10:02:07,943 [Thread-11] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
  89. 2012-08-28 10:02:07,943 [Thread-11] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
  90. 2012-08-28 10:02:07,943 [Thread-11] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
  91. 2012-08-28 10:02:08,008 [Thread-12] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : null
  92. 2012-08-28 10:02:08,011 [Thread-12] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed file:/tmp/temp382990916/tmp-2005021410/part-r-00000:0+1037
  93. 2012-08-28 10:02:08,012 [Thread-12] INFO org.apache.hadoop.mapred.MapTask - io.sort.mb = 100
  94. 2012-08-28 10:02:08,023 [Thread-12] INFO org.apache.hadoop.mapred.MapTask - data buffer = 79691776/99614720
  95. 2012-08-28 10:02:08,023 [Thread-12] INFO org.apache.hadoop.mapred.MapTask - record buffer = 262144/327680
  96. 2012-08-28 10:02:08,036 [Thread-12] INFO org.apache.hadoop.mapred.MapTask - Starting flush of map output
  97. 2012-08-28 10:02:08,047 [Thread-12] INFO org.apache.hadoop.mapred.MapTask - Finished spill 0
  98. 2012-08-28 10:02:08,048 [Thread-12] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0002_m_000000_0 is done. And is in the process of commiting
  99. 2012-08-28 10:02:08,391 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0002
  100. 2012-08-28 10:02:11,006 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner -
  101. 2012-08-28 10:02:11,006 [Thread-12] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0002_m_000000_0' done.
  102. 2012-08-28 10:02:11,013 [Thread-12] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : null
  103. 2012-08-28 10:02:11,013 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner -
  104. 2012-08-28 10:02:11,014 [Thread-12] INFO org.apache.hadoop.mapred.Merger - Merging 1 sorted segments
  105. 2012-08-28 10:02:11,015 [Thread-12] INFO org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 1 segments left of total size: 809 bytes
  106. 2012-08-28 10:02:11,015 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner -
  107. 2012-08-28 10:02:11,028 [Thread-12] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0002_r_000000_0 is done. And is in the process of commiting
  108. 2012-08-28 10:02:11,030 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner -
  109. 2012-08-28 10:02:11,030 [Thread-12] INFO org.apache.hadoop.mapred.Task - Task attempt_local_0002_r_000000_0 is allowed to commit now
  110. 2012-08-28 10:02:11,032 [Thread-12] INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local_0002_r_000000_0' to file:/Users/ceteri/src/concur/Impatient/part4/output/wc
  111. 2012-08-28 10:02:14,009 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce
  112. 2012-08-28 10:02:14,010 [Thread-12] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0002_r_000000_0' done.
  113. 2012-08-28 10:02:14,011 [Thread-12] WARN org.apache.hadoop.mapred.FileOutputCommitter - Output path is null in cleanup
  114. 2012-08-28 10:02:18,409 [main] WARN org.apache.pig.tools.pigstats.PigStatsUtil - Failed to get RunningJob for job job_local_0002
  115. 2012-08-28 10:02:18,411 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
  116. 2012-08-28 10:02:18,413 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
  117.  
  118. HadoopVersion PigVersion UserId StartedAt FinishedAt Features
  119. 1.0.3 0.10.0 ceteri 2012-08-28 10:01:49 2012-08-28 10:02:18 HASH_JOIN,GROUP_BY,FILTER
  120.  
  121. Success!
  122.  
  123. Job Stats (time in seconds):
  124. JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs
  125. job_local_0001 1 1 n/a n/a n/a n/a n/a n/a docPipe,stopPipe,tokenPipe HASH_JOIN
  126. job_local_0002 1 1 n/a n/a n/a n/a n/a n/a tokenGroups,wcPipe GROUP_BY,COMBINER file:///Users/ceteri/src/concur/Impatient/part4/output/wc,
  127.  
  128. Input(s):
  129. Successfully read 0 records from: "file:///Users/ceteri/src/concur/Impatient/part4/data/rain.txt"
  130. Successfully read 0 records from: "file:///Users/ceteri/src/concur/Impatient/part4/data/en.stop"
  131.  
  132. Output(s):
  133. Successfully stored 0 records in: "file:///Users/ceteri/src/concur/Impatient/part4/output/wc"
  134.  
  135. Counters:
  136. Total records written : 0
  137. Total bytes written : 0
  138. Spillable Memory Manager spill count : 0
  139. Total bags proactively spilled: 0
  140. Total records proactively spilled: 0
  141.  
  142. Job DAG:
  143. job_local_0001 -> job_local_0002,
  144. job_local_0002
  145.  
  146.  
  147. 2012-08-28 10:02:18,413 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
  148. bash-3.2$ cat output/wc/part-r-00000
  149. air 1
  150. dry 3
  151. dvd 1
  152. lee 2
  153. area 4
  154. land 2
  155. less 1
  156. lies 1
  157. rain 5
  158. such 1
  159. cause 1
  160. death 1
  161. known 1
  162. women 1
  163. broken 1
  164. effect 1
  165. ranges 1
  166. shadow 4
  167. valley 1
  168. deserts 1
  169. leeward 2
  170. primary 1
  171. secrets 1
  172. sinking 1
  173. downwind 1
  174. mountain 3
  175. produces 1
  176. australia 1
  177. cloudcover 1
  178. mountainous 1
  179. california's 1
  180. bash-3.2$
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!