Pastebin launched a little side project called VERYVIRAL.com, check it out ;-) Want more features on Pastebin? Sign Up, it's FREE!
Guest

Untitled

By: a guest on May 23rd, 2012  |  syntax: None  |  size: 13.26 KB  |  views: 21  |  expires: Never
download  |  raw  |  embed  |  report abuse  |  print
Text below is selected. Please press Ctrl+C to copy to your clipboard. (⌘+C on Mac)
  1. 2012-05-23 11:27:17,923 [main] INFO  org.apache.pig.Main - Apache Pig version 0.11.0-SNAPSHOT (rexportiert) compiled Mai 23 2012, 10:32:21
  2. 2012-05-23 11:27:17,924 [main] INFO  org.apache.pig.Main - Logging error messages to: /home/schwenk/Desktop/pig-debug/pig_1337765237921.log
  3. 2012-05-23 11:27:18,104 [main] INFO  org.apache.hadoop.security.UserGroupInformation - JAAS Configuration already set up for Hadoop, not re-installing.
  4. 2012-05-23 11:27:18,161 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /home/schwenk/.pigbootup not found
  5. 2012-05-23 11:27:18,238 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
  6. #-----------------------------------------------
  7. # New Logical Plan:
  8. #-----------------------------------------------
  9. b: (Name: LOStore Schema: id#10:int,grp#11:int,additional#12:int,referer#13:chararray)
  10. |
  11. |---b: (Name: LOFilter Schema: id#10:int,grp#11:int,additional#12:int,referer#13:chararray)
  12.     |   |
  13.     |   (Name: Or Type: boolean Uid: 23)
  14.     |   |
  15.     |   |---(Name: UserFunc(com.adition.pig.filtering.string.CONTAINS) Type: boolean Uid: 20)
  16.     |   |   |
  17.     |   |   |---referer:(Name: Project Type: chararray Uid: 13 Input: 0 Column: 3)
  18.     |   |   |
  19.     |   |   |---(Name: Constant Type: chararray Uid: 19)
  20.     |   |
  21.     |   |---(Name: UserFunc(com.adition.pig.filtering.string.CONTAINS) Type: boolean Uid: 22)
  22.     |       |
  23.     |       |---referer:(Name: Project Type: chararray Uid: 13 Input: 0 Column: 3)
  24.     |       |
  25.     |       |---(Name: Constant Type: chararray Uid: 21)
  26.     |
  27.     |---a: (Name: LOForEach Schema: id#10:int,grp#11:int,additional#12:int,referer#13:chararray)
  28.         |   |
  29.         |   (Name: LOGenerate[false,false,false,false] Schema: id#10:int,grp#11:int,additional#12:int,referer#13:chararray)ColumnPrune:InputUids=[10, 11, 12, 13]ColumnPrune:OutputUids=[10, 11, 12, 13]
  30.         |   |   |
  31.         |   |   (Name: Cast Type: int Uid: 10)
  32.         |   |   |
  33.         |   |   |---id:(Name: Project Type: bytearray Uid: 10 Input: 0 Column: (*))
  34.         |   |   |
  35.         |   |   (Name: Cast Type: int Uid: 11)
  36.         |   |   |
  37.         |   |   |---grp:(Name: Project Type: bytearray Uid: 11 Input: 1 Column: (*))
  38.         |   |   |
  39.         |   |   (Name: Cast Type: int Uid: 12)
  40.         |   |   |
  41.         |   |   |---additional:(Name: Project Type: bytearray Uid: 12 Input: 2 Column: (*))
  42.         |   |   |
  43.         |   |   (Name: Cast Type: chararray Uid: 13)
  44.         |   |   |
  45.         |   |   |---referer:(Name: Project Type: bytearray Uid: 13 Input: 3 Column: (*))
  46.         |   |
  47.         |   |---(Name: LOInnerLoad[0] Schema: id#10:bytearray)
  48.         |   |
  49.         |   |---(Name: LOInnerLoad[1] Schema: grp#11:bytearray)
  50.         |   |
  51.         |   |---(Name: LOInnerLoad[2] Schema: additional#12:bytearray)
  52.         |   |
  53.         |   |---(Name: LOInnerLoad[3] Schema: referer#13:bytearray)
  54.         |
  55.         |---a: (Name: LOLoad Schema: id#10:bytearray,grp#11:bytearray,additional#12:bytearray,referer#13:bytearray)RequiredFields:null
  56.  
  57. #-----------------------------------------------
  58. # Physical Plan:
  59. #-----------------------------------------------
  60. b: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-22
  61. |
  62. |---b: Filter[bag] - scope-14
  63.     |   |
  64.     |   Or[boolean] - scope-21
  65.     |   |
  66.     |   |---POUserFunc(com.adition.pig.filtering.string.CONTAINS)[boolean] - scope-17
  67.     |   |   |
  68.     |   |   |---Project[chararray][3] - scope-15
  69.     |   |   |
  70.     |   |   |---Constant(obama) - scope-16
  71.     |   |
  72.     |   |---POUserFunc(com.adition.pig.filtering.string.CONTAINS)[boolean] - scope-20
  73.     |       |
  74.     |       |---Project[chararray][3] - scope-18
  75.     |       |
  76.     |       |---Constant(praesident) - scope-19
  77.     |
  78.     |---a: New For Each(false,false,false,false)[bag] - scope-13
  79.         |   |
  80.         |   Cast[int] - scope-2
  81.         |   |
  82.         |   |---Project[bytearray][0] - scope-1
  83.         |   |
  84.         |   Cast[int] - scope-5
  85.         |   |
  86.         |   |---Project[bytearray][1] - scope-4
  87.         |   |
  88.         |   Cast[int] - scope-8
  89.         |   |
  90.         |   |---Project[bytearray][2] - scope-7
  91.         |   |
  92.         |   Cast[chararray] - scope-11
  93.         |   |
  94.         |   |---Project[bytearray][3] - scope-10
  95.         |
  96.         |---a: Load(file:///home/schwenk/Desktop/pig-debug/TestCONTAINS-testFilteringCluster-input.txt:org.apache.pig.builtin.PigStorage) - scope-0
  97.  
  98. 2012-05-23 11:27:18,979 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
  99. 2012-05-23 11:27:19,011 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
  100. 2012-05-23 11:27:19,011 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
  101. #--------------------------------------------------
  102. # Map Reduce Plan                                  
  103. #--------------------------------------------------
  104. MapReduce node scope-23
  105. Map Plan
  106. b: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-22
  107. |
  108. |---b: Filter[bag] - scope-14
  109.     |   |
  110.     |   Or[boolean] - scope-21
  111.     |   |
  112.     |   |---POUserFunc(com.adition.pig.filtering.string.CONTAINS)[boolean] - scope-17
  113.     |   |   |
  114.     |   |   |---Project[chararray][3] - scope-15
  115.     |   |   |
  116.     |   |   |---Constant(obama) - scope-16
  117.     |   |
  118.     |   |---POUserFunc(com.adition.pig.filtering.string.CONTAINS)[boolean] - scope-20
  119.     |       |
  120.     |       |---Project[chararray][3] - scope-18
  121.     |       |
  122.     |       |---Constant(praesident) - scope-19
  123.     |
  124.     |---a: New For Each(false,false,false,false)[bag] - scope-13
  125.         |   |
  126.         |   Cast[int] - scope-2
  127.         |   |
  128.         |   |---Project[bytearray][0] - scope-1
  129.         |   |
  130.         |   Cast[int] - scope-5
  131.         |   |
  132.         |   |---Project[bytearray][1] - scope-4
  133.         |   |
  134.         |   Cast[int] - scope-8
  135.         |   |
  136.         |   |---Project[bytearray][2] - scope-7
  137.         |   |
  138.         |   Cast[chararray] - scope-11
  139.         |   |
  140.         |   |---Project[bytearray][3] - scope-10
  141.         |
  142.         |---a: Load(file:///home/schwenk/Desktop/pig-debug/TestCONTAINS-testFilteringCluster-input.txt:org.apache.pig.builtin.PigStorage) - scope-0--------
  143. Global sort: false
  144. ----------------
  145.  
  146. 2012-05-23 11:27:19,061 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: FILTER
  147. 2012-05-23 11:27:19,085 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
  148. 2012-05-23 11:27:19,088 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
  149. 2012-05-23 11:27:19,088 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
  150. 2012-05-23 11:27:19,099 [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
  151. 2012-05-23 11:27:19,111 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
  152. 2012-05-23 11:27:19,129 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
  153. 2012-05-23 11:27:19,134 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
  154. 2012-05-23 11:27:19,136 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=505
  155. 2012-05-23 11:27:19,136 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
  156. 2012-05-23 11:27:19,159 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
  157. 2012-05-23 11:27:19,210 [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
  158. 2012-05-23 11:27:19,211 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
  159. 2012-05-23 11:27:19,239 [Thread-5] INFO  org.apache.hadoop.util.NativeCodeLoader - Loaded the native-hadoop library
  160. 2012-05-23 11:27:19,265 [Thread-5] WARN  org.apache.hadoop.mapred.JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
  161. 2012-05-23 11:27:19,304 [Thread-5] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
  162. 2012-05-23 11:27:19,305 [Thread-5] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
  163. 2012-05-23 11:27:19,315 [Thread-5] WARN  org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library is available
  164. 2012-05-23 11:27:19,315 [Thread-5] INFO  org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library loaded
  165. 2012-05-23 11:27:19,318 [Thread-5] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
  166. 2012-05-23 11:27:19,597 [Thread-6] INFO  org.apache.hadoop.util.ProcessTree - setsid exited with exit code 0
  167. 2012-05-23 11:27:19,603 [Thread-6] INFO  org.apache.hadoop.mapred.Task -  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@c45aa2c
  168. 2012-05-23 11:27:19,616 [Thread-6] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed file:/home/schwenk/Desktop/pig-debug/TestCONTAINS-testFilteringCluster-input.txt:0+505
  169. 2012-05-23 11:27:19,641 [Thread-6] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map - Aliases being processed per job phase (AliasName[line,offset]): M: a[3,4],a[-1,-1],b[4,4] C:  R:
  170. 2012-05-23 11:27:19,646 [Thread-6] INFO  org.apache.hadoop.mapred.Task - Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
  171. 2012-05-23 11:27:19,650 [Thread-6] INFO  org.apache.hadoop.mapred.LocalJobRunner -
  172. 2012-05-23 11:27:19,650 [Thread-6] INFO  org.apache.hadoop.mapred.Task - Task attempt_local_0001_m_000000_0 is allowed to commit now
  173. 2012-05-23 11:27:19,653 [Thread-6] INFO  org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local_0001_m_000000_0' to file:/tmp/temp-390375712/tmp-1337884501
  174. 2012-05-23 11:27:19,654 [Thread-6] INFO  org.apache.hadoop.mapred.LocalJobRunner -
  175. 2012-05-23 11:27:19,654 [Thread-6] INFO  org.apache.hadoop.mapred.Task - Task 'attempt_local_0001_m_000000_0' done.
  176. 2012-05-23 11:27:19,655 [Thread-6] WARN  org.apache.hadoop.mapred.FileOutputCommitter - Output path is null in cleanup
  177. 2012-05-23 11:27:19,712 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001
  178. 2012-05-23 11:27:19,712 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases a,b
  179. 2012-05-23 11:27:19,712 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: a[3,4],a[-1,-1],b[4,4] C:  R:
  180. 2012-05-23 11:27:19,718 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
  181. 2012-05-23 11:27:20,226 [main] WARN  org.apache.pig.tools.pigstats.PigStatsUtil - Failed to get RunningJob for job job_local_0001
  182. 2012-05-23 11:27:20,234 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
  183. 2012-05-23 11:27:20,235 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats reported below may be incomplete
  184. 2012-05-23 11:27:20,240 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
  185.  
  186. HadoopVersion   PigVersion      UserId  StartedAt       FinishedAt      Features
  187. 0.20.2-cdh3u3   0.11.0-SNAPSHOT schwenk 2012-05-23 11:27:19     2012-05-23 11:27:20     FILTER
  188.  
  189. Success!
  190.  
  191. Job Stats (time in seconds):
  192. JobId   Alias   Feature Outputs
  193. job_local_0001  a,b     MAP_ONLY        file:/tmp/temp-390375712/tmp-1337884501,
  194.  
  195. Input(s):
  196. Successfully read records from: "file:///home/schwenk/Desktop/pig-debug/TestCONTAINS-testFilteringCluster-input.txt"
  197.  
  198. Output(s):
  199. Successfully stored records in: "file:/tmp/temp-390375712/tmp-1337884501"
  200.  
  201. Job DAG:
  202. job_local_0001
  203.  
  204.  
  205. 2012-05-23 11:27:20,243 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
  206. 2012-05-23 11:27:20,248 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
  207. 2012-05-23 11:27:20,248 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
  208. (4,323,242,http://www.google.com/url&url=http%3A%2F%2Fwww.tagesschau.de&q=obama)
  209. (5,423,342,http://www.google.com/url&url=http%3A%2F%2Fwww.bild.de&q=obama)
  210. (6,523,442,http://www.google.com/url&url=http%3A%2F%2Fwww.example.com%2Fmypage.htm&q=praesident)