Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- 2012-05-23 11:30:57,620 [main] INFO org.apache.pig.Main - Logging error messages to: /home/schwenk/Desktop/pig-debug/pig_1337765457616.log
- 2012-05-23 11:30:57,743 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
- 2012-05-23 11:30:57,874 [main] INFO org.apache.hadoop.security.UserGroupInformation - JAAS Configuration already set up for Hadoop, not re-installing.
- 2012-05-23 11:30:58,076 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - pig.usenewlogicalplan is set to true. New logical plan will be used.
- #-----------------------------------------------
- # Logical Plan:
- #-----------------------------------------------
- fake: Store 1-18 Schema: {id: int,grp: int,additional: int,referer: chararray} Type: Unknown
- |
- |---b: Filter 1-17 Schema: {id: int,grp: int,additional: int,referer: chararray} Type: bag
- | |
- | Or 1-16 FieldSchema: boolean Type: boolean
- | |
- | |---UserFunc 1-12 function: com.adition.pig.filtering.string.CONTAINS FieldSchema: com.adition.pig.filtering.string.contains_referer_1: boolean Type: boolean
- | | |
- | | |---Project 1-10 Projections: [3] Overloaded: false FieldSchema: referer: chararray Type: chararray
- | | | Input: a: Load 1-9
- | | |
- | | |---Const 1-11( obama ) FieldSchema: chararray Type: chararray
- | |
- | |---UserFunc 1-15 function: com.adition.pig.filtering.string.CONTAINS FieldSchema: com.adition.pig.filtering.string.contains_referer_2: boolean Type: boolean
- | |
- | |---Project 1-13 Projections: [3] Overloaded: false FieldSchema: referer: chararray Type: chararray
- | | Input: a: Load 1-9
- | |
- | |---Const 1-14( praesident ) FieldSchema: chararray Type: chararray
- |
- |---a: Load 1-9 Schema: {id: int,grp: int,additional: int,referer: chararray} Type: bag
- #-----------------------------------------------
- # New Logical Plan:
- #-----------------------------------------------
- fake: (Name: LOStore Schema: id#14:int,grp#15:int,additional#16:int,referer#17:chararray)
- |
- |---b: (Name: LOFilter Schema: id#14:int,grp#15:int,additional#16:int,referer#17:chararray)
- | |
- | (Name: UserFunc(com.adition.pig.filtering.string.CONTAINS) Type: boolean Uid: 19)
- | |
- | |---referer:(Name: Project Type: chararray Uid: 17 Input: 0 Column: 3)
- | |
- | |---(Name: Constant Type: chararray Uid: 18)
- |
- |---a: (Name: LOForEach Schema: id#14:int,grp#15:int,additional#16:int,referer#17:chararray)
- | |
- | (Name: LOGenerate[false,false,false,false] Schema: id#14:int,grp#15:int,additional#16:int,referer#17:chararray)ColumnPrune:InputUids=[17, 16, 14, 15]ColumnPrune:OutputUids=[17, 16, 14, 15]
- | | |
- | | (Name: Cast Type: int Uid: 14)
- | | |
- | | |---id:(Name: Project Type: bytearray Uid: 14 Input: 0 Column: 0)
- | | |
- | | (Name: Cast Type: int Uid: 15)
- | | |
- | | |---grp:(Name: Project Type: bytearray Uid: 15 Input: 1 Column: 0)
- | | |
- | | (Name: Cast Type: int Uid: 16)
- | | |
- | | |---additional:(Name: Project Type: bytearray Uid: 16 Input: 2 Column: 0)
- | | |
- | | (Name: Cast Type: chararray Uid: 17)
- | | |
- | | |---referer:(Name: Project Type: bytearray Uid: 17 Input: 3 Column: 0)
- | |
- | |---(Name: LOInnerLoad[0] Schema: id#14:bytearray)
- | |
- | |---(Name: LOInnerLoad[1] Schema: grp#15:bytearray)
- | |
- | |---(Name: LOInnerLoad[2] Schema: additional#16:bytearray)
- | |
- | |---(Name: LOInnerLoad[3] Schema: referer#17:bytearray)
- |
- |---a: (Name: LOLoad Schema: id#14:bytearray,grp#15:bytearray,additional#16:bytearray,referer#17:bytearray)RequiredFields:null
- #-----------------------------------------------
- # Physical Plan:
- #-----------------------------------------------
- b: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-18
- |
- |---b: Filter[bag] - scope-14
- | |
- | POUserFunc(com.adition.pig.filtering.string.CONTAINS)[boolean] - scope-17
- | |
- | |---Project[chararray][3] - scope-15
- | |
- | |---Constant(obama) - scope-16
- |
- |---a: New For Each(false,false,false,false)[bag] - scope-13
- | |
- | Cast[int] - scope-2
- | |
- | |---Project[bytearray][0] - scope-1
- | |
- | Cast[int] - scope-5
- | |
- | |---Project[bytearray][1] - scope-4
- | |
- | Cast[int] - scope-8
- | |
- | |---Project[bytearray][2] - scope-7
- | |
- | Cast[chararray] - scope-11
- | |
- | |---Project[bytearray][3] - scope-10
- |
- |---a: Load(file:///home/schwenk/Desktop/pig-debug/TestCONTAINS-testFilteringCluster-input.txt:org.apache.pig.builtin.PigStorage) - scope-0
- 2012-05-23 11:30:58,301 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
- 2012-05-23 11:30:58,327 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
- 2012-05-23 11:30:58,328 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
- #--------------------------------------------------
- # Map Reduce Plan
- #--------------------------------------------------
- MapReduce node scope-19
- Map Plan
- b: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-18
- |
- |---b: Filter[bag] - scope-14
- | |
- | POUserFunc(com.adition.pig.filtering.string.CONTAINS)[boolean] - scope-17
- | |
- | |---Project[chararray][3] - scope-15
- | |
- | |---Constant(obama) - scope-16
- |
- |---a: New For Each(false,false,false,false)[bag] - scope-13
- | |
- | Cast[int] - scope-2
- | |
- | |---Project[bytearray][0] - scope-1
- | |
- | Cast[int] - scope-5
- | |
- | |---Project[bytearray][1] - scope-4
- | |
- | Cast[int] - scope-8
- | |
- | |---Project[bytearray][2] - scope-7
- | |
- | Cast[chararray] - scope-11
- | |
- | |---Project[bytearray][3] - scope-10
- |
- |---a: Load(file:///home/schwenk/Desktop/pig-debug/TestCONTAINS-testFilteringCluster-input.txt:org.apache.pig.builtin.PigStorage) - scope-0--------
- Global sort: false
- ----------------
- 2012-05-23 11:30:58,358 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: FILTER
- 2012-05-23 11:30:58,358 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - pig.usenewlogicalplan is set to true. New logical plan will be used.
- 2012-05-23 11:30:58,409 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: b: Store(file:/tmp/temp642485280/tmp-673425882:org.apache.pig.impl.io.InterStorage) - scope-38 Operator Key: scope-38)
- 2012-05-23 11:30:58,409 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
- 2012-05-23 11:30:58,411 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
- 2012-05-23 11:30:58,411 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
- 2012-05-23 11:30:58,427 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
- 2012-05-23 11:30:58,437 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
- 2012-05-23 11:30:58,456 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
- 2012-05-23 11:31:00,178 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
- 2012-05-23 11:31:00,223 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
- 2012-05-23 11:31:00,224 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
- 2012-05-23 11:31:00,241 [Thread-2] INFO org.apache.hadoop.util.NativeCodeLoader - Loaded the native-hadoop library
- 2012-05-23 11:31:00,378 [Thread-2] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
- 2012-05-23 11:31:00,378 [Thread-2] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
- 2012-05-23 11:31:00,393 [Thread-2] WARN org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library is available
- 2012-05-23 11:31:00,393 [Thread-2] INFO org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library loaded
- 2012-05-23 11:31:00,396 [Thread-2] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
- 2012-05-23 11:31:00,686 [Thread-3] INFO org.apache.hadoop.util.ProcessTree - setsid exited with exit code 0
- 2012-05-23 11:31:00,692 [Thread-3] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@54fedfb6
- 2012-05-23 11:31:00,725 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001
- 2012-05-23 11:31:00,725 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
- 2012-05-23 11:31:00,740 [Thread-3] INFO org.apache.hadoop.mapred.Task - Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
- 2012-05-23 11:31:00,746 [Thread-3] INFO org.apache.hadoop.mapred.LocalJobRunner -
- 2012-05-23 11:31:00,747 [Thread-3] INFO org.apache.hadoop.mapred.Task - Task attempt_local_0001_m_000000_0 is allowed to commit now
- 2012-05-23 11:31:00,750 [Thread-3] INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local_0001_m_000000_0' to file:/tmp/temp642485280/tmp-673425882
- 2012-05-23 11:31:00,752 [Thread-3] INFO org.apache.hadoop.mapred.LocalJobRunner -
- 2012-05-23 11:31:00,753 [Thread-3] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local_0001_m_000000_0' done.
- 2012-05-23 11:31:05,733 [main] WARN org.apache.pig.tools.pigstats.PigStatsUtil - Failed to get RunningJob for job job_local_0001
- 2012-05-23 11:31:05,736 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
- 2012-05-23 11:31:05,736 [main] INFO org.apache.pig.tools.pigstats.PigStats - Detected Local mode. Stats reported below may be incomplete
- 2012-05-23 11:31:05,738 [main] INFO org.apache.pig.tools.pigstats.PigStats - Script Statistics:
- HadoopVersion PigVersion UserId StartedAt FinishedAt Features
- 0.20.2-cdh3u3 0.8.1-cdh3u3 schwenk 2012-05-23 11:30:58 2012-05-23 11:31:05 FILTER
- Success!
- Job Stats (time in seconds):
- JobId Alias Feature Outputs
- job_local_0001 a,b MAP_ONLY file:/tmp/temp642485280/tmp-673425882,
- Input(s):
- Successfully read records from: "file:///home/schwenk/Desktop/pig-debug/TestCONTAINS-testFilteringCluster-input.txt"
- Output(s):
- Successfully stored records in: "file:/tmp/temp642485280/tmp-673425882"
- Job DAG:
- job_local_0001
- 2012-05-23 11:31:05,739 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
- 2012-05-23 11:31:05,744 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
- 2012-05-23 11:31:05,744 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
- (4,323,242,http://www.google.com/url&url=http%3A%2F%2Fwww.tagesschau.de&q=obama)
- (5,423,342,http://www.google.com/url&url=http%3A%2F%2Fwww.bild.de&q=obama)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement