Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- Attempt one - the job fails
- [spry@hadoop2 giraph]$ hadoop jar giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner -Dgiraph.keepZooKeeperData=true -Dgiraph.cleanupCheckpointsAfterSuccess=false -Dgiraph.logLevel=DEBUG -Dgiraph.SplitMasterWorker=false -Dgiraph.zkList="localhost:2181" -Dgiraph.zkSessionMsecTimeout=600000 -Dgiraph.useInputSplitLocality=false org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/spry/input -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/spry/Shortest -w 1
- SLF4J: Class path contains multiple SLF4J bindings.
- SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/giraph-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
- SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
- 14/01/13 17:10:15 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one.
- 14/01/13 17:10:15 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one.
- 14/01/13 17:10:16 INFO yarn.GiraphYarnClient: Final output path is: hdfs://hadoop2.j7.master:8020/user/spry/Shortest
- 14/01/13 17:10:16 INFO yarn.GiraphYarnClient: Running Client
- 14/01/13 17:10:16 INFO client.RMProxy: Connecting to ResourceManager at hadoop2.j7.master/127.0.0.1:8050
- 14/01/13 17:10:16 INFO yarn.GiraphYarnClient: Got node report from ASM for, nodeId=localhost:45454, nodeAddresslocalhost:8042, nodeRackName/default-rack, nodeNumContainers0
- 14/01/13 17:10:16 INFO yarn.GiraphYarnClient: Defaulting per-task heap size to 512MB.
- 14/01/13 17:10:16 INFO yarn.GiraphYarnClient: Obtained new Application ID: application_1389643303411_0026
- 14/01/13 17:10:16 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
- 14/01/13 17:10:16 INFO yarn.GiraphYarnClient: Set the environment for the application master
- 14/01/13 17:10:16 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPATH=${CLASSPATH}:./*:/etc/hadoop/conf:/usr/lib/hadoop/*:/usr/lib/hadoop/lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-yarn/*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-mapreduce/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
- 14/01/13 17:10:16 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
- 14/01/13 17:10:16 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
- 14/01/13 17:10:16 INFO yarn.YarnUtils: Registered file in LocalResources :: hdfs://hadoop2.j7.master:8020/user/spry/giraph_yarn_jar_cache/application_1389643303411_0026/giraph-conf.xml
- 14/01/13 17:10:16 INFO yarn.GiraphYarnClient: LIB JARS :
- 14/01/13 17:10:16 INFO yarn.YarnUtils: Class path name .
- 14/01/13 17:10:16 INFO yarn.YarnUtils: base path checking .
- 14/01/13 17:10:17 INFO yarn.YarnUtils: Class path name null
- 14/01/13 17:10:17 INFO yarn.YarnUtils: base path checking null
- 14/01/13 17:10:17 INFO yarn.YarnUtils: Class path name /etc/hadoop/conf
- 14/01/13 17:10:17 INFO yarn.YarnUtils: base path checking /etc/hadoop/conf
- 14/01/13 17:10:17 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop/lib/*
- 14/01/13 17:10:17 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop/lib
- 14/01/13 17:10:17 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop/.//*
- 14/01/13 17:10:17 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop/./
- 14/01/13 17:10:17 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop-hdfs/./
- 14/01/13 17:10:17 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop-hdfs/./
- 14/01/13 17:10:17 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop-hdfs/lib/*
- 14/01/13 17:10:17 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop-hdfs/lib
- 14/01/13 17:10:17 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop-hdfs/.//*
- 14/01/13 17:10:17 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop-hdfs/./
- 14/01/13 17:10:17 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop-yarn/lib/*
- 14/01/13 17:10:17 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop-yarn/lib
- 14/01/13 17:10:17 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop-yarn/.//*
- 14/01/13 17:10:17 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop-yarn/./
- 14/01/13 17:10:17 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop-mapreduce/lib/*
- 14/01/13 17:10:17 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop-mapreduce/lib
- 14/01/13 17:10:17 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop-mapreduce/.//*
- 14/01/13 17:10:17 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop-mapreduce/./
- 14/01/13 17:10:17 INFO yarn.YarnUtils: Class path name
- 14/01/13 17:10:17 INFO yarn.YarnUtils: Class path name /usr/share/java/mysql-connector-java-5.1.17.jar
- 14/01/13 17:10:17 INFO yarn.YarnUtils: base path checking /usr/share/java/mysql-connector-java-5.1.17.jar
- 14/01/13 17:10:17 INFO yarn.YarnUtils: Class path name /usr/share/java/mysql-connector-java.jar
- 14/01/13 17:10:17 INFO yarn.YarnUtils: base path checking /usr/share/java/mysql-connector-java.jar
- 14/01/13 17:10:17 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop-mapreduce/*
- 14/01/13 17:10:17 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop-mapreduce
- 14/01/13 17:10:17 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated.
- 14/01/13 17:10:17 INFO yarn.GiraphYarnClient: Submitting application to ASM
- 14/01/13 17:10:17 INFO impl.YarnClientImpl: Submitted application application_1389643303411_0026 to ResourceManager at hadoop2.j7.master/127.0.0.1:8050
- 14/01/13 17:10:17 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1389643303411_0026
- 14/01/13 17:10:17 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation
- 14/01/13 17:10:17 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.83 secs
- 14/01/13 17:10:17 INFO yarn.GiraphYarnClient: appattempt_1389643303411_0026_000001, State: ACCEPTED, Containers used: 1
- 14/01/13 17:10:21 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 4.84 secs
- 14/01/13 17:10:21 INFO yarn.GiraphYarnClient: appattempt_1389643303411_0026_000001, State: ACCEPTED, Containers used: 1
- 14/01/13 17:10:26 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 8.88 secs
- 14/01/13 17:10:26 INFO yarn.GiraphYarnClient: appattempt_1389643303411_0026_000001, State: RUNNING, Containers used: 3
- 14/01/13 17:10:30 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 12.89 secs
- 14/01/13 17:10:30 INFO yarn.GiraphYarnClient: appattempt_1389643303411_0026_000001, State: RUNNING, Containers used: 3
- 14/01/13 17:10:34 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 16.91 secs
- 14/01/13 17:10:34 INFO yarn.GiraphYarnClient: appattempt_1389643303411_0026_000001, State: RUNNING, Containers used: 3
- 14/01/13 17:10:38 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 20.92 secs
- 14/01/13 17:10:38 INFO yarn.GiraphYarnClient: appattempt_1389643303411_0026_000001, State: RUNNING, Containers used: 3
- 14/01/13 17:10:39 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed cache directory for Giraph job.
- 14/01/13 17:10:39 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total running time: 0 minutes, 22 seconds.
- Remove the created output directory
- [spry@hadoop2 giraph]$ hadoop fs -rmr /user/spry/Shortest
- rmr: DEPRECATED: Please use 'rm -r' instead.
- SLF4J: Class path contains multiple SLF4J bindings.
- SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/giraph-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
- SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
- 14/01/13 17:10:49 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 21600000 minutes, Emptier interval = 0 minutes.
- Moved: 'hdfs://hadoop2.j7.master:8020/user/spry/Shortest' to trash at: hdfs://hadoop2.j7.master:8020/user/spry/.Trash/Current
- Attempt two - no changes to the command at all. the job succeeds
- [spry@hadoop2 giraph]$ hadoop jar giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner -Dgiraph.keepZooKeeperData=true -Dgiraph.cleanupCheckpointsAfterSuccess=false -Dgiraph.logLevel=DEBUG -Dgiraph.SplitMasterWorker=false -Dgiraph.zkList="localhost:2181" -Dgiraph.zkSessionMsecTimeout=600000 -Dgiraph.useInputSplitLocality=false org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/spry/input -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/spry/Shortest -w 1
- SLF4J: Class path contains multiple SLF4J bindings.
- SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/giraph-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
- SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
- SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
- 14/01/13 17:11:00 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one.
- 14/01/13 17:11:00 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one.
- 14/01/13 17:11:00 INFO yarn.GiraphYarnClient: Final output path is: hdfs://hadoop2.j7.master:8020/user/spry/Shortest
- 14/01/13 17:11:00 INFO yarn.GiraphYarnClient: Running Client
- 14/01/13 17:11:00 INFO client.RMProxy: Connecting to ResourceManager at hadoop2.j7.master/127.0.0.1:8050
- 14/01/13 17:11:01 INFO yarn.GiraphYarnClient: Got node report from ASM for, nodeId=localhost:45454, nodeAddresslocalhost:8042, nodeRackName/default-rack, nodeNumContainers0
- 14/01/13 17:11:01 INFO yarn.GiraphYarnClient: Defaulting per-task heap size to 512MB.
- 14/01/13 17:11:01 INFO yarn.GiraphYarnClient: Obtained new Application ID: application_1389643303411_0027
- 14/01/13 17:11:01 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
- 14/01/13 17:11:01 INFO yarn.GiraphYarnClient: Set the environment for the application master
- 14/01/13 17:11:01 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPATH=${CLASSPATH}:./*:/etc/hadoop/conf:/usr/lib/hadoop/*:/usr/lib/hadoop/lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-yarn/*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-mapreduce/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
- 14/01/13 17:11:01 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
- 14/01/13 17:11:01 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
- 14/01/13 17:11:01 INFO yarn.YarnUtils: Registered file in LocalResources :: hdfs://hadoop2.j7.master:8020/user/spry/giraph_yarn_jar_cache/application_1389643303411_0027/giraph-conf.xml
- 14/01/13 17:11:01 INFO yarn.GiraphYarnClient: LIB JARS :
- 14/01/13 17:11:01 INFO yarn.YarnUtils: Class path name .
- 14/01/13 17:11:01 INFO yarn.YarnUtils: base path checking .
- 14/01/13 17:11:02 INFO yarn.YarnUtils: Class path name null
- 14/01/13 17:11:02 INFO yarn.YarnUtils: base path checking null
- 14/01/13 17:11:02 INFO yarn.YarnUtils: Class path name /etc/hadoop/conf
- 14/01/13 17:11:02 INFO yarn.YarnUtils: base path checking /etc/hadoop/conf
- 14/01/13 17:11:02 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop/lib/*
- 14/01/13 17:11:02 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop/lib
- 14/01/13 17:11:02 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop/.//*
- 14/01/13 17:11:02 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop/./
- 14/01/13 17:11:02 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop-hdfs/./
- 14/01/13 17:11:02 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop-hdfs/./
- 14/01/13 17:11:02 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop-hdfs/lib/*
- 14/01/13 17:11:02 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop-hdfs/lib
- 14/01/13 17:11:02 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop-hdfs/.//*
- 14/01/13 17:11:02 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop-hdfs/./
- 14/01/13 17:11:02 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop-yarn/lib/*
- 14/01/13 17:11:02 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop-yarn/lib
- 14/01/13 17:11:02 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop-yarn/.//*
- 14/01/13 17:11:02 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop-yarn/./
- 14/01/13 17:11:02 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop-mapreduce/lib/*
- 14/01/13 17:11:02 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop-mapreduce/lib
- 14/01/13 17:11:02 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop-mapreduce/.//*
- 14/01/13 17:11:02 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop-mapreduce/./
- 14/01/13 17:11:02 INFO yarn.YarnUtils: Class path name
- 14/01/13 17:11:02 INFO yarn.YarnUtils: Class path name /usr/share/java/mysql-connector-java-5.1.17.jar
- 14/01/13 17:11:02 INFO yarn.YarnUtils: base path checking /usr/share/java/mysql-connector-java-5.1.17.jar
- 14/01/13 17:11:02 INFO yarn.YarnUtils: Class path name /usr/share/java/mysql-connector-java.jar
- 14/01/13 17:11:02 INFO yarn.YarnUtils: base path checking /usr/share/java/mysql-connector-java.jar
- 14/01/13 17:11:02 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop-mapreduce/*
- 14/01/13 17:11:02 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop-mapreduce
- 14/01/13 17:11:02 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated.
- 14/01/13 17:11:02 INFO yarn.GiraphYarnClient: Submitting application to ASM
- 14/01/13 17:11:02 INFO impl.YarnClientImpl: Submitted application application_1389643303411_0027 to ResourceManager at hadoop2.j7.master/127.0.0.1:8050
- 14/01/13 17:11:02 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1389643303411_0027
- 14/01/13 17:11:02 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation
- 14/01/13 17:11:02 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.83 secs
- 14/01/13 17:11:02 INFO yarn.GiraphYarnClient: appattempt_1389643303411_0027_000001, State: ACCEPTED, Containers used: 0
- 14/01/13 17:11:06 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 4.85 secs
- 14/01/13 17:11:06 INFO yarn.GiraphYarnClient: appattempt_1389643303411_0027_000001, State: ACCEPTED, Containers used: 1
- 14/01/13 17:11:10 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 8.86 secs
- 14/01/13 17:11:10 INFO yarn.GiraphYarnClient: appattempt_1389643303411_0027_000001, State: RUNNING, Containers used: 1
- 14/01/13 17:11:14 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 12.87 secs
- 14/01/13 17:11:14 INFO yarn.GiraphYarnClient: appattempt_1389643303411_0027_000001, State: RUNNING, Containers used: 3
- 14/01/13 17:11:18 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 16.89 secs
- 14/01/13 17:11:18 INFO yarn.GiraphYarnClient: appattempt_1389643303411_0027_000001, State: RUNNING, Containers used: 3
- 14/01/13 17:11:22 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 20.90 secs
- 14/01/13 17:11:22 INFO yarn.GiraphYarnClient: appattempt_1389643303411_0027_000001, State: RUNNING, Containers used: 3
- 14/01/13 17:11:26 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed cache directory for Giraph job.
- 14/01/13 17:11:27 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation: SUCCEEDED, total running time: 0 minutes, 24 seconds.
Advertisement
Add Comment
Please, Sign In to add comment