API tools faq
paste
Advertisement
Guest User

Untitled

a guest
May 23rd, 2012
59
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 13.26 KB | None | 0 0
raw download clone embed print report
  1. 2012-05-23 11:27:17,923 [main] INFO org.apache.pig.Main - Apache Pig version 0.11.0-SNAPSHOT (rexportiert) compiled Mai 23 2012, 10:32:21
  2. 2012-05-23 11:27:17,924 [main] INFO org.apache.pig.Main - Logging error messages to: /home/schwenk/Desktop/pig-debug/pig_1337765237921.log
  3. 2012-05-23 11:27:18,104 [main] INFO org.apache.hadoop.security.UserGroupInformation - JAAS Configuration already set up for Hadoop, not re-installing.
  4. 2012-05-23 11:27:18,161 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/schwenk/.pigbootup not found
  5. 2012-05-23 11:27:18,238 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
  6. #-----------------------------------------------
  7. # New Logical Plan:
  8. #-----------------------------------------------
  9. b: (Name: LOStore Schema: id#10:int,grp#11:int,additional#12:int,referer#13:chararray)
  10. |
  11. |---b: (Name: LOFilter Schema: id#10:int,grp#11:int,additional#12:int,referer#13:chararray)
  12. | |
  13. | (Name: Or Type: boolean Uid: 23)
  14. | |
  15. | |---(Name: UserFunc(com.adition.pig.filtering.string.CONTAINS) Type: boolean Uid: 20)
  16. | | |
  17. | | |---referer:(Name: Project Type: chararray Uid: 13 Input: 0 Column: 3)
  18. | | |
  19. | | |---(Name: Constant Type: chararray Uid: 19)
  20. | |
  21. | |---(Name: UserFunc(com.adition.pig.filtering.string.CONTAINS) Type: boolean Uid: 22)
  22. | |
  23. | |---referer:(Name: Project Type: chararray Uid: 13 Input: 0 Column: 3)
  24. | |
  25. | |---(Name: Constant Type: chararray Uid: 21)
  26. |
  27. |---a: (Name: LOForEach Schema: id#10:int,grp#11:int,additional#12:int,referer#13:chararray)
  28. | |
  29. | (Name: LOGenerate[false,false,false,false] Schema: id#10:int,grp#11:int,additional#12:int,referer#13:chararray)ColumnPrune:InputUids=[10, 11, 12, 13]ColumnPrune:OutputUids=[10, 11, 12, 13]
  30. | | |
  31. | | (Name: Cast Type: int Uid: 10)
  32. | | |
  33. | | |---id:(Name: Project Type: bytearray Uid: 10 Input: 0 Column: (*))
  34. | | |
  35. | | (Name: Cast Type: int Uid: 11)
  36. | | |
  37. | | |---grp:(Name: Project Type: bytearray Uid: 11 Input: 1 Column: (*))
  38. | | |
  39. | | (Name: Cast Type: int Uid: 12)
  40. | | |
  41. | | |---additional:(Name: Project Type: bytearray Uid: 12 Input: 2 Column: (*))
  42. | | |
  43. | | (Name: Cast Type: chararray Uid: 13)
  44. | | |
  45. | | |---referer:(Name: Project Type: bytearray Uid: 13 Input: 3 Column: (*))
  46. | |
  47. | |---(Name: LOInnerLoad[0] Schema: id#10:bytearray)
  48. | |
  49. | |---(Name: LOInnerLoad[1] Schema: grp#11:bytearray)
  50. | |
  51. | |---(Name: LOInnerLoad[2] Schema: additional#12:bytearray)
  52. | |
  53. | |---(Name: LOInnerLoad[3] Schema: referer#13:bytearray)
  54. |
  55. |---a: (Name: LOLoad Schema: id#10:bytearray,grp#11:bytearray,additional#12:bytearray,referer#13:bytearray)RequiredFields:null
  56.  
  57. #-----------------------------------------------
  58. # Physical Plan:
  59. #-----------------------------------------------
  60. b: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-22
  61. |
  62. |---b: Filter[bag] - scope-14
  63. | |
  64. | Or[boolean] - scope-21
  65. | |
  66. | |---POUserFunc(com.adition.pig.filtering.string.CONTAINS)[boolean] - scope-17
  67. | | |
  68. | | |---Project[chararray][3] - scope-15
  69. | | |
  70. | | |---Constant(obama) - scope-16
  71. | |
  72. | |---POUserFunc(com.adition.pig.filtering.string.CONTAINS)[boolean] - scope-20
  73. | |
  74. | |---Project[chararray][3] - scope-18
  75. | |
  76. | |---Constant(praesident) - scope-19
  77. |
  78. |---a: New For Each(false,false,false,false)[bag] - scope-13
  79. | |
  80. | Cast[int] - scope-2
  81. | |
  82. | |---Project[bytearray][0] - scope-1
  83. | |
  84. | Cast[int] - scope-5
  85. | |
  86. | |---Project[bytearray][1] - scope-4
  87. | |
  88. | Cast[int] - scope-8
  89. | |
  90. | |---Project[bytearray][2] - scope-7
  91. | |
  92. | Cast[chararray] - scope-11
  93. | |
  94. | |---Project[bytearray][3] - scope-10
  95. |
  96. |---a: Load(file:///home/schwenk/Desktop/pig-debug/TestCONTAINS-testFilteringCluster-input.txt:org.apache.pig.builtin.PigStorage) - scope-0
  97.  
  98. 2012-05-23 11:27:18,979 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
  99. 2012-05-23 11:27:19,011 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
  100. 2012-05-23 11:27:19,011 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
  101. #--------------------------------------------------
  102. # Map Reduce Plan
  103. #--------------------------------------------------
  104. MapReduce node scope-23
  105. Map Plan
  106. b: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-22
  107. |
  108. |---b: Filter[bag] - scope-14
  109. | |
  110. | Or[boolean] - scope-21
  111. | |
  112. | |---POUserFunc(com.adition.pig.filtering.string.CONTAINS)[boolean] - scope-17
  113. | | |
  114. | | |---Project[chararray][3] - scope-15
  115. | | |
  116. | | |---Constant(obama) - scope-16
  117. | |
  118. | |---POUserFunc(com.adition.pig.filtering.string.CONTAINS)[boolean] - scope-20
  119. | |
  120. | |---Project[chararray][3] - scope-18
  121. | |
  122. | |---Constant(praesident) - scope-19
  123. |
  124. |---a: New For Each(false,false,false,false)[bag] - scope-13
  125. | |
  126. | Cast[int] - scope-2
  127. | |
  128. | |---Project[bytearray][0] - scope-1
  129. | |
  130. | Cast[int] - scope-5
  131. | |
  132. | |---Project[bytearray][1] - scope-4
  133. | |
  134. | Cast[int] - scope-8
  135. | |
  136. | |---Project[bytearray][2] - scope-7
  137. | |
  138. | Cast[chararray] - scope-11
  139. | |
  140. | |---Project[bytearray][3] - scope-10
  141. |
  142. |---a: Load(file:///home/schwenk/Desktop/pig-debug/TestCONTAINS-testFilteringCluster-input.txt:org.apache.pig.builtin.PigStorage) - scope-0--------
  143. Global sort: false
  144. ----------------
  145.  
  146. 2012-05-23 11:27:19,061 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: FILTER
  147. 2012-05-23 11:27:19,085 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
  148. 2012-05-23 11:27:19,088 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
  149. 2012-05-23 11:27:19,088 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
  150. 2012-05-23 11:27:19,099 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
  151. 2012-05-23 11:27:19,111 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
  152. 2012-05-23 11:27:19,129 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
  153. 2012-05-23 11:27:19,134 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
  154. 2012-05-23 11:27:19,136 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=505
  155. 2012-05-23 11:27:19,136 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
  156. 2012-05-23 11:27:19,159 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
  157. 2012-05-23 11:27:19,210 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
  158. 2012-05-23 11:27:19,211 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
  159. 2012-05-23 11:27:19,239 [Thread-5] INFO org.apache.hadoop.util.NativeCodeLoader - Loaded the native-hadoop library
  160. 2012-05-23 11:27:19,265 [Thread-5] WARN org.apache.hadoop.mapred.JobClient - No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
  161. 2012-05-23 11:27:19,304 [Thread-5] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
  162. 2012-05-23 11:27:19,305 [Thread-5] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
  163. 2012-05-23 11:27:19,315 [Thread-5] WARN org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library is available
  164. 2012-05-23 11:27:19,315 [Thread-5] INFO org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library loaded
  165. 2012-05-23 11:27:19,318 [Thread-5] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
  166. 2012-05-23 11:27:19,597 [Thread-6] INFO org.apache.hadoop.util.ProcessTree - setsid exited with exit code 0
  167. 2012-05-23 11:27:19,603 [Thread-6] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@c45aa2c
  168. 2012-05-23 11:27:19,616 [Thread-6] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed file:/home/schwenk/Desktop/pig-debug/TestCONTAINS-testFilteringCluster-input.txt:0+505
  169. 2012-05-23 11:27:19,641 [Thread-6] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map - Aliases being processed per job phase (AliasName[line,offset]): M: a[3,4],a[-1,-1],b[4,4] C: R:
  170. 2012-05-23 11:27:19,646 [Thread-6] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
  171. 2012-05-23 11:27:19,650 [Thread-6] INFO org.apache.hadoop.mapred.LocalJobRunner -
  172. 2012-05-23 11:27:19,650 [Thread-6] INFO org.apache.hadoop.mapred.Task - Task attempt_local_0001_m_000000_0 is allowed to commit now
  173. 2012-05-23 11:27:19,653 [Thread-6] INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local_0001_m_000000_0' to file:/tmp/temp-390375712/tmp-1337884501
  174. 2012-05-23 11:27:19,654 [Thread-6] INFO org.apache.hadoop.mapred.LocalJobRunner -
  175. 2012-05-23 11:27:19,654 [Thread-6] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0001_m_000000_0' done.
  176. 2012-05-23 11:27:19,655 [Thread-6] WARN org.apache.hadoop.mapred.FileOutputCommitter - Output path is null in cleanup
  177. 2012-05-23 11:27:19,712 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001
  178. 2012-05-23 11:27:19,712 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases a,b
  179. 2012-05-23 11:27:19,712 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: a[3,4],a[-1,-1],b[4,4] C: R:
  180. 2012-05-23 11:27:19,718 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
  181. 2012-05-23 11:27:20,226 [main] WARN org.apache.pig.tools.pigstats.PigStatsUtil - Failed to get RunningJob for job job_local_0001
  182. 2012-05-23 11:27:20,234 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
  183. 2012-05-23 11:27:20,235 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats reported below may be incomplete
  184. 2012-05-23 11:27:20,240 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
  185.  
  186. HadoopVersion PigVersion UserId StartedAt FinishedAt Features
  187. 0.20.2-cdh3u3 0.11.0-SNAPSHOT schwenk 2012-05-23 11:27:19 2012-05-23 11:27:20 FILTER
  188.  
  189. Success!
  190.  
  191. Job Stats (time in seconds):
  192. JobId Alias Feature Outputs
  193. job_local_0001 a,b MAP_ONLY file:/tmp/temp-390375712/tmp-1337884501,
  194.  
  195. Input(s):
  196. Successfully read records from: "file:///home/schwenk/Desktop/pig-debug/TestCONTAINS-testFilteringCluster-input.txt"
  197.  
  198. Output(s):
  199. Successfully stored records in: "file:/tmp/temp-390375712/tmp-1337884501"
  200.  
  201. Job DAG:
  202. job_local_0001
  203.  
  204.  
  205. 2012-05-23 11:27:20,243 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
  206. 2012-05-23 11:27:20,248 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
  207. 2012-05-23 11:27:20,248 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
  208. (4,323,242,http://www.google.com/url&url=http%3A%2F%2Fwww.tagesschau.de&q=obama)
  209. (5,423,342,http://www.google.com/url&url=http%3A%2F%2Fwww.bild.de&q=obama)
  210. (6,523,442,http://www.google.com/url&url=http%3A%2F%2Fwww.example.com%2Fmypage.htm&q=praesident)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!