Pastebin launched a little side project called VERYVIRAL.com, check it out ;-) Want more features on Pastebin? Sign Up, it's FREE!
Guest

Untitled

By: a guest on May 23rd, 2012  |  syntax: None  |  size: 12.24 KB  |  views: 17  |  expires: Never
download  |  raw  |  embed  |  report abuse  |  print
Text below is selected. Please press Ctrl+C to copy to your clipboard. (⌘+C on Mac)
  1. 2012-05-23 11:30:57,620 [main] INFO  org.apache.pig.Main - Logging error messages to: /home/schwenk/Desktop/pig-debug/pig_1337765457616.log
  2. 2012-05-23 11:30:57,743 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
  3. 2012-05-23 11:30:57,874 [main] INFO  org.apache.hadoop.security.UserGroupInformation - JAAS Configuration already set up for Hadoop, not re-installing.
  4. 2012-05-23 11:30:58,076 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - pig.usenewlogicalplan is set to true. New logical plan will be used.
  5. #-----------------------------------------------
  6. # Logical Plan:
  7. #-----------------------------------------------
  8. fake: Store 1-18 Schema: {id: int,grp: int,additional: int,referer: chararray} Type: Unknown
  9. |
  10. |---b: Filter 1-17 Schema: {id: int,grp: int,additional: int,referer: chararray} Type: bag
  11.     |   |
  12.     |   Or 1-16 FieldSchema: boolean Type: boolean
  13.     |   |
  14.     |   |---UserFunc 1-12 function: com.adition.pig.filtering.string.CONTAINS FieldSchema: com.adition.pig.filtering.string.contains_referer_1: boolean Type: boolean
  15.     |   |   |
  16.     |   |   |---Project 1-10 Projections: [3] Overloaded: false FieldSchema: referer: chararray Type: chararray
  17.     |   |   |   Input: a: Load 1-9
  18.     |   |   |
  19.     |   |   |---Const 1-11( obama ) FieldSchema: chararray Type: chararray
  20.     |   |
  21.     |   |---UserFunc 1-15 function: com.adition.pig.filtering.string.CONTAINS FieldSchema: com.adition.pig.filtering.string.contains_referer_2: boolean Type: boolean
  22.     |       |
  23.     |       |---Project 1-13 Projections: [3] Overloaded: false FieldSchema: referer: chararray Type: chararray
  24.     |       |   Input: a: Load 1-9
  25.     |       |
  26.     |       |---Const 1-14( praesident ) FieldSchema: chararray Type: chararray
  27.     |
  28.     |---a: Load 1-9 Schema: {id: int,grp: int,additional: int,referer: chararray} Type: bag
  29.  
  30. #-----------------------------------------------
  31. # New Logical Plan:
  32. #-----------------------------------------------
  33. fake: (Name: LOStore Schema: id#14:int,grp#15:int,additional#16:int,referer#17:chararray)
  34. |
  35. |---b: (Name: LOFilter Schema: id#14:int,grp#15:int,additional#16:int,referer#17:chararray)
  36.     |   |
  37.     |   (Name: UserFunc(com.adition.pig.filtering.string.CONTAINS) Type: boolean Uid: 19)
  38.     |   |
  39.     |   |---referer:(Name: Project Type: chararray Uid: 17 Input: 0 Column: 3)
  40.     |   |
  41.     |   |---(Name: Constant Type: chararray Uid: 18)
  42.     |
  43.     |---a: (Name: LOForEach Schema: id#14:int,grp#15:int,additional#16:int,referer#17:chararray)
  44.         |   |
  45.         |   (Name: LOGenerate[false,false,false,false] Schema: id#14:int,grp#15:int,additional#16:int,referer#17:chararray)ColumnPrune:InputUids=[17, 16, 14, 15]ColumnPrune:OutputUids=[17, 16, 14, 15]
  46.         |   |   |
  47.         |   |   (Name: Cast Type: int Uid: 14)
  48.         |   |   |
  49.         |   |   |---id:(Name: Project Type: bytearray Uid: 14 Input: 0 Column: 0)
  50.         |   |   |
  51.         |   |   (Name: Cast Type: int Uid: 15)
  52.         |   |   |
  53.         |   |   |---grp:(Name: Project Type: bytearray Uid: 15 Input: 1 Column: 0)
  54.         |   |   |
  55.         |   |   (Name: Cast Type: int Uid: 16)
  56.         |   |   |
  57.         |   |   |---additional:(Name: Project Type: bytearray Uid: 16 Input: 2 Column: 0)
  58.         |   |   |
  59.         |   |   (Name: Cast Type: chararray Uid: 17)
  60.         |   |   |
  61.         |   |   |---referer:(Name: Project Type: bytearray Uid: 17 Input: 3 Column: 0)
  62.         |   |
  63.         |   |---(Name: LOInnerLoad[0] Schema: id#14:bytearray)
  64.         |   |
  65.         |   |---(Name: LOInnerLoad[1] Schema: grp#15:bytearray)
  66.         |   |
  67.         |   |---(Name: LOInnerLoad[2] Schema: additional#16:bytearray)
  68.         |   |
  69.         |   |---(Name: LOInnerLoad[3] Schema: referer#17:bytearray)
  70.         |
  71.         |---a: (Name: LOLoad Schema: id#14:bytearray,grp#15:bytearray,additional#16:bytearray,referer#17:bytearray)RequiredFields:null
  72.  
  73. #-----------------------------------------------
  74. # Physical Plan:
  75. #-----------------------------------------------
  76. b: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-18
  77. |
  78. |---b: Filter[bag] - scope-14
  79.     |   |
  80.     |   POUserFunc(com.adition.pig.filtering.string.CONTAINS)[boolean] - scope-17
  81.     |   |
  82.     |   |---Project[chararray][3] - scope-15
  83.     |   |
  84.     |   |---Constant(obama) - scope-16
  85.     |
  86.     |---a: New For Each(false,false,false,false)[bag] - scope-13
  87.         |   |
  88.         |   Cast[int] - scope-2
  89.         |   |
  90.         |   |---Project[bytearray][0] - scope-1
  91.         |   |
  92.         |   Cast[int] - scope-5
  93.         |   |
  94.         |   |---Project[bytearray][1] - scope-4
  95.         |   |
  96.         |   Cast[int] - scope-8
  97.         |   |
  98.         |   |---Project[bytearray][2] - scope-7
  99.         |   |
  100.         |   Cast[chararray] - scope-11
  101.         |   |
  102.         |   |---Project[bytearray][3] - scope-10
  103.         |
  104.         |---a: Load(file:///home/schwenk/Desktop/pig-debug/TestCONTAINS-testFilteringCluster-input.txt:org.apache.pig.builtin.PigStorage) - scope-0
  105.  
  106. 2012-05-23 11:30:58,301 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
  107. 2012-05-23 11:30:58,327 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
  108. 2012-05-23 11:30:58,328 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
  109. #--------------------------------------------------
  110. # Map Reduce Plan                                  
  111. #--------------------------------------------------
  112. MapReduce node scope-19
  113. Map Plan
  114. b: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-18
  115. |
  116. |---b: Filter[bag] - scope-14
  117.     |   |
  118.     |   POUserFunc(com.adition.pig.filtering.string.CONTAINS)[boolean] - scope-17
  119.     |   |
  120.     |   |---Project[chararray][3] - scope-15
  121.     |   |
  122.     |   |---Constant(obama) - scope-16
  123.     |
  124.     |---a: New For Each(false,false,false,false)[bag] - scope-13
  125.         |   |
  126.         |   Cast[int] - scope-2
  127.         |   |
  128.         |   |---Project[bytearray][0] - scope-1
  129.         |   |
  130.         |   Cast[int] - scope-5
  131.         |   |
  132.         |   |---Project[bytearray][1] - scope-4
  133.         |   |
  134.         |   Cast[int] - scope-8
  135.         |   |
  136.         |   |---Project[bytearray][2] - scope-7
  137.         |   |
  138.         |   Cast[chararray] - scope-11
  139.         |   |
  140.         |   |---Project[bytearray][3] - scope-10
  141.         |
  142.         |---a: Load(file:///home/schwenk/Desktop/pig-debug/TestCONTAINS-testFilteringCluster-input.txt:org.apache.pig.builtin.PigStorage) - scope-0--------
  143. Global sort: false
  144. ----------------
  145.  
  146. 2012-05-23 11:30:58,358 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: FILTER
  147. 2012-05-23 11:30:58,358 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - pig.usenewlogicalplan is set to true. New logical plan will be used.
  148. 2012-05-23 11:30:58,409 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: b: Store(file:/tmp/temp642485280/tmp-673425882:org.apache.pig.impl.io.InterStorage) - scope-38 Operator Key: scope-38)
  149. 2012-05-23 11:30:58,409 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
  150. 2012-05-23 11:30:58,411 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
  151. 2012-05-23 11:30:58,411 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
  152. 2012-05-23 11:30:58,427 [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
  153. 2012-05-23 11:30:58,437 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
  154. 2012-05-23 11:30:58,456 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
  155. 2012-05-23 11:31:00,178 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
  156. 2012-05-23 11:31:00,223 [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
  157. 2012-05-23 11:31:00,224 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
  158. 2012-05-23 11:31:00,241 [Thread-2] INFO  org.apache.hadoop.util.NativeCodeLoader - Loaded the native-hadoop library
  159. 2012-05-23 11:31:00,378 [Thread-2] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
  160. 2012-05-23 11:31:00,378 [Thread-2] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
  161. 2012-05-23 11:31:00,393 [Thread-2] WARN  org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library is available
  162. 2012-05-23 11:31:00,393 [Thread-2] INFO  org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library loaded
  163. 2012-05-23 11:31:00,396 [Thread-2] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
  164. 2012-05-23 11:31:00,686 [Thread-3] INFO  org.apache.hadoop.util.ProcessTree - setsid exited with exit code 0
  165. 2012-05-23 11:31:00,692 [Thread-3] INFO  org.apache.hadoop.mapred.Task -  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@54fedfb6
  166. 2012-05-23 11:31:00,725 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001
  167. 2012-05-23 11:31:00,725 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
  168. 2012-05-23 11:31:00,740 [Thread-3] INFO  org.apache.hadoop.mapred.Task - Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
  169. 2012-05-23 11:31:00,746 [Thread-3] INFO  org.apache.hadoop.mapred.LocalJobRunner -
  170. 2012-05-23 11:31:00,747 [Thread-3] INFO  org.apache.hadoop.mapred.Task - Task attempt_local_0001_m_000000_0 is allowed to commit now
  171. 2012-05-23 11:31:00,750 [Thread-3] INFO  org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local_0001_m_000000_0' to file:/tmp/temp642485280/tmp-673425882
  172. 2012-05-23 11:31:00,752 [Thread-3] INFO  org.apache.hadoop.mapred.LocalJobRunner -
  173. 2012-05-23 11:31:00,753 [Thread-3] INFO  org.apache.hadoop.mapred.Task - Task 'attempt_local_0001_m_000000_0' done.
  174. 2012-05-23 11:31:05,733 [main] WARN  org.apache.pig.tools.pigstats.PigStatsUtil - Failed to get RunningJob for job job_local_0001
  175. 2012-05-23 11:31:05,736 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
  176. 2012-05-23 11:31:05,736 [main] INFO  org.apache.pig.tools.pigstats.PigStats - Detected Local mode. Stats reported below may be incomplete
  177. 2012-05-23 11:31:05,738 [main] INFO  org.apache.pig.tools.pigstats.PigStats - Script Statistics:
  178.  
  179. HadoopVersion   PigVersion      UserId  StartedAt       FinishedAt      Features
  180. 0.20.2-cdh3u3   0.8.1-cdh3u3    schwenk 2012-05-23 11:30:58     2012-05-23 11:31:05     FILTER
  181.  
  182. Success!
  183.  
  184. Job Stats (time in seconds):
  185. JobId   Alias   Feature Outputs
  186. job_local_0001  a,b     MAP_ONLY        file:/tmp/temp642485280/tmp-673425882,
  187.  
  188. Input(s):
  189. Successfully read records from: "file:///home/schwenk/Desktop/pig-debug/TestCONTAINS-testFilteringCluster-input.txt"
  190.  
  191. Output(s):
  192. Successfully stored records in: "file:/tmp/temp642485280/tmp-673425882"
  193.  
  194. Job DAG:
  195. job_local_0001
  196.  
  197.  
  198. 2012-05-23 11:31:05,739 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
  199. 2012-05-23 11:31:05,744 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
  200. 2012-05-23 11:31:05,744 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
  201. (4,323,242,http://www.google.com/url&url=http%3A%2F%2Fwww.tagesschau.de&q=obama)
  202. (5,423,342,http://www.google.com/url&url=http%3A%2F%2Fwww.bild.de&q=obama)
clone this paste RAW Paste Data