API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Feb 23rd, 2017
134
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 10.93 KB | None | 0 0
raw download clone embed print report
  1. SLF4J: Class path contains multiple SLF4J bindings.
  2. SLF4J: Found binding in [jar:file:/data/tmp/nm-local-dir/usercache/hadoop/filecache/25/__spark_libs__6843186552468170225.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  3. SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  4. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  5. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  6. 17/02/24 11:28:07 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 23988@datanode03-bi-dev
  7. 17/02/24 11:28:07 INFO SignalUtils: Registered signal handler for TERM
  8. 17/02/24 11:28:07 INFO SignalUtils: Registered signal handler for HUP
  9. 17/02/24 11:28:07 INFO SignalUtils: Registered signal handler for INT
  10. 17/02/24 11:28:07 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  11. 17/02/24 11:28:07 INFO SecurityManager: Changing view acls to: hadoop
  12. 17/02/24 11:28:07 INFO SecurityManager: Changing modify acls to: hadoop
  13. 17/02/24 11:28:07 INFO SecurityManager: Changing view acls groups to:
  14. 17/02/24 11:28:07 INFO SecurityManager: Changing modify acls groups to:
  15. 17/02/24 11:28:07 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
  16. 17/02/24 11:28:08 INFO TransportClientFactory: Successfully created connection to /1111.111.111.111:37395 after 106 ms (0 ms spent in bootstraps)
  17. 17/02/24 11:28:08 INFO SecurityManager: Changing view acls to: hadoop
  18. 17/02/24 11:28:08 INFO SecurityManager: Changing modify acls to: hadoop
  19. 17/02/24 11:28:08 INFO SecurityManager: Changing view acls groups to:
  20. 17/02/24 11:28:08 INFO SecurityManager: Changing modify acls groups to:
  21. 17/02/24 11:28:08 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
  22. 17/02/24 11:28:08 INFO TransportClientFactory: Successfully created connection to /1111.111.111.111:37395 after 2 ms (0 ms spent in bootstraps)
  23. 17/02/24 11:28:08 INFO DiskBlockManager: Created local directory at /data/tmp/nm-local-dir/usercache/hadoop/appcache/application_1487905765621_0005/blockmgr-a20a4b37-66f0-4fc7-bc8a-879935ab100c
  24. 17/02/24 11:28:08 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
  25. 17/02/24 11:28:08 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@1111.111.111.111:37395
  26. 17/02/24 11:28:08 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
  27. 17/02/24 11:28:08 INFO Executor: Starting executor ID 3 on host datanode03-bi-dev
  28. 17/02/24 11:28:08 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 60029.
  29. 17/02/24 11:28:08 INFO NettyBlockTransferService: Server created on datanode03-bi-dev:60029
  30. 17/02/24 11:28:08 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(3, datanode03-bi-dev, 60029)
  31. 17/02/24 11:28:08 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(3, datanode03-bi-dev, 60029)
  32. 17/02/24 11:28:09 INFO CoarseGrainedExecutorBackend: Got assigned task 2
  33. 17/02/24 11:28:09 INFO Executor: Running task 2.0 in stage 0.0 (TID 2)
  34. 17/02/24 11:28:10 INFO TorrentBroadcast: Started reading broadcast variable 2
  35. 17/02/24 11:28:10 INFO TransportClientFactory: Successfully created connection to /1111.111.111.111:34436 after 1 ms (0 ms spent in bootstraps)
  36. 17/02/24 11:28:10 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 4.2 KB, free 912.3 MB)
  37. 17/02/24 11:28:10 INFO TorrentBroadcast: Reading broadcast variable 2 took 132 ms
  38. 17/02/24 11:28:10 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 5.9 KB, free 912.3 MB)
  39. 2017-02-24 11:28:20,601 INFO (MainThread-24059) TFSparkNode.reserve: {'authkey': UUID('1b726e7a-c176-4974-8cb7-a3984d54028f'), 'worker_num': 2, 'host': 'datanode03-bi-dev', 'tb_port': 0, 'addr': '/tmp/pymp-vMwVR6/listener-w9GOmI', 'ppid': 24048, 'task_index': 1, 'job_name': 'worker', 'tb_pid': 0, 'port': 27900}
  40. 17/02/24 11:28:20 INFO PythonRunner: Times: total = 10307, boot = 251, init = 30, finish = 10026
  41. 17/02/24 11:28:20 INFO Executor: Finished task 2.0 in stage 0.0 (TID 2). 2387 bytes result sent to driver
  42. 17/02/24 11:28:21 INFO CoarseGrainedExecutorBackend: Got assigned task 7
  43. 17/02/24 11:28:21 INFO Executor: Running task 2.0 in stage 1.0 (TID 7)
  44. 17/02/24 11:28:21 INFO TorrentBroadcast: Started reading broadcast variable 3
  45. tensorflow/core/distributed_runtime/master_session.cc:1012] Start master session ad78225c6eb5cd2d with config:
  46.  
  47. INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not17/02/24 11:28:21 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 6.3 KB, free 912.3 MB)
  48. 17/02/24 11:28:21 INFO TorrentBroadcast: Reading broadcast variable 3 took 16 ms
  49. 17/02/24 11:28:21 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 9.3 KB, free 912.3 MB)
  50. 2017-02-24 11:28:21,203 INFO (MainThread-24059) node: {'addr': ('datanode01-bi-dev', 12241), 'task_index': 0, 'job_name': 'ps', 'authkey': UUID('0843b53f-22a3-4cf5-ab0d-d316ba7f792c'), 'worker_num': 0, 'host': 'datanode01-bi-dev', 'ppid': 28724, 'port': 47797, 'tb_pid': 0, 'tb_port': 0}
  51. 2017-02-24 11:28:21,203 INFO (MainThread-24059) node: {'addr': '/tmp/pymp-jCE5VH/listener-zlzwwH', 'task_index': 0, 'job_name': 'worker', 'authkey': UUID('256af5f7-287a-4c4d-8165-9816bff21808'), 'worker_num': 1, 'host': 'namenode01-bi-dev', 'ppid': 25867, 'port': 59678, 'tb_pid': 0, 'tb_port': 0}
  52. 2017-02-24 11:28:21,203 INFO (MainThread-24059) node: {'addr': '/tmp/pymp-vMwVR6/listener-w9GOmI', 'task_index': 1, 'job_name': 'worker', 'authkey': UUID('1b726e7a-c176-4974-8cb7-a3984d54028f'), 'worker_num': 2, 'host': 'datanode03-bi-dev', 'ppid': 24048, 'port': 27900, 'tb_pid': 0, 'tb_port': 0}
  53. 2017-02-24 11:28:21,203 INFO (MainThread-24059) node: {'addr': '/tmp/pymp-WZymp4/listener-ilxHlD', 'task_index': 2, 'job_name': 'worker', 'authkey': UUID('fda214a3-a8ed-40cc-bf70-60a807475c50'), 'worker_num': 3, 'host': 'datanode02-bi-dev', 'ppid': 26047, 'port': 19377, 'tb_pid': 0, 'tb_port': 0}
  54. 2017-02-24 11:28:21,203 INFO (MainThread-24059) node: {'addr': '/tmp/pymp-4wlOfY/listener-6eigjl', 'task_index': 3, 'job_name': 'worker', 'authkey': UUID('cb7d84ce-8fd2-445d-a3d4-d43b7e8f9e8e'), 'worker_num': 4, 'host': 'datanode04-bi-dev', 'ppid': 2409, 'port': 13204, 'tb_pid': 0, 'tb_port': 0}
  55. 2017-02-24 11:28:21,210 INFO (MainThread-24059) Connected to TFSparkNode.mgr on datanode03-bi-dev, ppid=24048, state='running'
  56. 2017-02-24 11:28:21,394 INFO (MainThread-24059) Starting TensorFlow worker:1 on cluster node 2 on background thread
  57. 17/02/24 11:28:21 INFO PythonRunner: Times: total = 203, boot = -573, init = 582, finish = 194
  58. 17/02/24 11:28:21 INFO Executor: Finished task 2.0 in stage 1.0 (TID 7). 2078 bytes result sent to driver
  59. 2017-02-24 11:28:22,185 INFO (Thread-1-24059) 2: ======== worker:1 ========
  60. 2017-02-24 11:28:22,185 INFO (Thread-1-24059) 2: Cluster spec: {'ps': ['datanode01-bi-dev:47797'], 'worker': ['namenode01-bi-dev:59678', 'datanode03-bi-dev:27900', 'datanode02-bi-dev:19377', 'datanode04-bi-dev:13204']}
  61. 2017-02-24 11:28:22,185 INFO (Thread-1-24059) 2: Using CPU
  62. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
  63. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
  64. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
  65. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
  66. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
  67. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
  68. D0224 11:28:22.187804383 24116 env_linux.c:77] Warning: insecure environment read function 'getenv' used
  69. I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:200] Initialize GrpcChannelCache for job ps -> {0 -> datanode01-bi-dev:47797}
  70. I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:200] Initialize GrpcChannelCache for job worker -> {0 -> namenode01-bi-dev:59678, 1 -> localhost:27900, 2 -> datanode02-bi-dev:19377, 3 -> datanode04-bi-dev:13204}
  71. I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:221] Started server with target: grpc://localhost:27900
  72. tensorflow/core/distributed_runtime/master_session.cc:1012] Start master session 71d70f79630b507b with config:
  73.  
  74. 2017-02-24T11:28:57.626287 session ready
  75. 2017-02-24T11:28:57.743252 step: 0 accuracy: 0.469999998808
  76. 2017-02-24T11:28:59.866619 step: 100 accuracy: 0.949999988079
  77. 2017-02-24 11:29:00,893 INFO (MainThread-24154) Processed 6144 items in partition
  78. 17/02/24 11:29:00 INFO PythonRunner: Times: total = 33240, boot = 4, init = 32, finish = 33204
  79. 17/02/24 11:29:00 INFO Executor: Finished task 2.0 in stage 2.0 (TID 12). 2442 bytes result sent to driver
  80. 17/02/24 11:29:00 INFO CoarseGrainedExecutorBackend: Got assigned task 15
  81. 17/02/24 11:29:00 INFO Executor: Running task 6.0 in stage 2.0 (TID 15)
  82. 17/02/24 11:29:00 INFO HadoopRDD: Input split: hdfs://1111.111.111.111:8020/user/hadoop/mnist/csv/train/images/part-00006:0+11214285
  83. 17/02/24 11:29:01 INFO HadoopRDD: Input split: hdfs://1111.111.111.111:8020/user/hadoop/mnist/csv/train/labels/part-00006:0+245760
  84. 2017-02-24 11:29:01,128 INFO (MainThread-24154) Connected to TFSparkNode.mgr on datanode03-bi-dev, ppid=24048, state='running'
  85. 2017-02-24 11:29:01,136 INFO (MainThread-24154) mgr.state='running'
  86. 2017-02-24 11:29:01,136 INFO (MainThread-24154) Feeding partition <generator object load_stream at 0x7febe0a7a690> into input queue <multiprocessing.queues.JoinableQueue object at 0x7febe0a711d0>
  87. 17/02/24 11:29:05 INFO PythonRunner: Times: total = 4116, boot = -30780, init = 30786, finish = 4110
  88. 17/02/24 11:29:05 INFO PythonRunner: Times: total = 96, boot = -33274, init = 33295, finish = 75
  89. 2017-02-24 11:29:08,041 INFO (MainThread-24154) Processed 6144 items in partitiontensorflow model pa
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!