API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Feb 23rd, 2017
125
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 15.44 KB | None | 0 0
raw download clone embed print report
  1. SLF4J: Class path contains multiple SLF4J bindings.
  2. SLF4J: Found binding in [jar:file:/data/tmp/nm-local-dir/usercache/hadoop/filecache/26/__spark_libs__6843186552468170225.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  3. SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  4. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  5. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  6. 17/02/24 11:28:07 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 25988@datanode02-bi-dev
  7. 17/02/24 11:28:07 INFO SignalUtils: Registered signal handler for TERM
  8. 17/02/24 11:28:07 INFO SignalUtils: Registered signal handler for HUP
  9. 17/02/24 11:28:07 INFO SignalUtils: Registered signal handler for INT
  10. 17/02/24 11:28:08 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  11. 17/02/24 11:28:08 INFO SecurityManager: Changing view acls to: hadoop
  12. 17/02/24 11:28:08 INFO SecurityManager: Changing modify acls to: hadoop
  13. 17/02/24 11:28:08 INFO SecurityManager: Changing view acls groups to:
  14. 17/02/24 11:28:08 INFO SecurityManager: Changing modify acls groups to:
  15. 17/02/24 11:28:08 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
  16. 17/02/24 11:28:09 INFO TransportClientFactory: Successfully created connection to /ipipip:37395 after 85 ms (0 ms spent in bootstraps)
  17. 17/02/24 11:28:09 INFO SecurityManager: Changing view acls to: hadoop
  18. 17/02/24 11:28:09 INFO SecurityManager: Changing modify acls to: hadoop
  19. 17/02/24 11:28:09 INFO SecurityManager: Changing view acls groups to:
  20. 17/02/24 11:28:09 INFO SecurityManager: Changing modify acls groups to:
  21. 17/02/24 11:28:09 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
  22. 17/02/24 11:28:09 INFO TransportClientFactory: Successfully created connection to /ipipip:37395 after 2 ms (0 ms spent in bootstraps)
  23. 17/02/24 11:28:09 INFO DiskBlockManager: Created local directory at /data/tmp/nm-local-dir/usercache/hadoop/appcache/application_1487905765621_0005/blockmgr-52842ffd-d363-4fbe-ba50-05e2d8e912db
  24. 17/02/24 11:28:09 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
  25. 17/02/24 11:28:09 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@ipipip:37395
  26. 17/02/24 11:28:09 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
  27. 17/02/24 11:28:09 INFO Executor: Starting executor ID 5 on host datanode02-bi-dev
  28. 17/02/24 11:28:09 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 8868.
  29. 17/02/24 11:28:09 INFO NettyBlockTransferService: Server created on datanode02-bi-dev:8868
  30. 17/02/24 11:28:09 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(5, datanode02-bi-dev, 8868)
  31. 17/02/24 11:28:09 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(5, datanode02-bi-dev, 8868)
  32. 17/02/24 11:28:09 INFO CoarseGrainedExecutorBackend: Got assigned task 3
  33. 17/02/24 11:28:09 INFO Executor: Running task 3.0 in stage 0.0 (TID 3)
  34. 17/02/24 11:28:10 INFO TorrentBroadcast: Started reading broadcast variable 2
  35. 17/02/24 11:28:10 INFO TransportClientFactory: Successfully created connection to /ipipip:34436 after 2 ms (0 ms spent in bootstraps)
  36. 17/02/24 11:28:10 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 4.2 KB, free 912.3 MB)
  37. 17/02/24 11:28:10 INFO TorrentBroadcast: Reading broadcast variable 2 took 117 ms
  38. 17/02/24 11:28:10 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 5.9 KB, free 912.3 MB)
  39. 2017-02-24 11:28:20,621 INFO (MainThread-26057) TFSparkNode.reserve: {'authkey': UUID('fda214a3-a8ed-40cc-bf70-60a807475c50'), 'worker_num': 3, 'host': 'datanode02-bi-dev', 'tb_port': 0, 'addr': '/tmp/pymp-WZymp4/listener-ilxHlD', 'ppid': 26047, 'task_index': 2, 'job_name': 'worker', 'tb_pid': 0, 'port': 19377}
  40. 17/02/24 11:28:20 INFO PythonRunner: Times: total = 10360, boot = 293, init = 37, finish = 10030
  41. 17/02/24 11:28:20 INFO Executor: Finished task 3.0 in stage 0.0 (TID 3). 2315 bytes result sent to driver
  42. 17/02/24 11:28:21 INFO CoarseGrainedExecutorBackend: Got assigned task 9
  43. 17/02/24 11:28:21 INFO Executor: Running task 4.0 in stage 1.0 (TID 9)
  44. 17/02/24 11:28:21 INFO TorrentBroadcast: Started reading broadcast variable 3
  45. 17/02/24 11:28:21 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 6.3 KB, free 912.3 MB)
  46. 17/02/24 11:28:21 INFO TorrentBroadcast: Reading broadcast variable 3 took 21 ms
  47. 17/02/24 11:28:21 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 9.3 KB, free 912.3 MB)
  48. 2017-02-24 11:28:21,200 INFO (MainThread-26057) node: {'addr': ('datanode01-bi-dev', 12241), 'task_index': 0, 'job_name': 'ps', 'authkey': UUID('0843b53f-22a3-4cf5-ab0d-d316ba7f792c'), 'worker_num': 0, 'host': 'datanode01-bi-dev', 'ppid': 28724, 'port': 47797, 'tb_pid': 0, 'tb_port': 0}
  49. 2017-02-24 11:28:21,200 INFO (MainThread-26057) node: {'addr': '/tmp/pymp-jCE5VH/listener-zlzwwH', 'task_index': 0, 'job_name': 'worker', 'authkey': UUID('256af5f7-287a-4c4d-8165-9816bff21808'), 'worker_num': 1, 'host': 'namenode01-bi-dev', 'ppid': 25867, 'port': 59678, 'tb_pid': 0, 'tb_port': 0}
  50. 2017-02-24 11:28:21,201 INFO (MainThread-26057) node: {'addr': '/tmp/pymp-vMwVR6/listener-w9GOmI', 'task_index': 1, 'job_name': 'worker', 'authkey': UUID('1b726e7a-c176-4974-8cb7-a3984d54028f'), 'worker_num': 2, 'host': 'datanode03-bi-dev', 'ppid': 24048, 'port': 27900, 'tb_pid': 0, 'tb_port': 0}
  51. 2017-02-24 11:28:21,201 INFO (MainThread-26057) node: {'addr': '/tmp/pymp-WZymp4/listener-ilxHlD', 'task_index': 2, 'job_name': 'worker', 'authkey': UUID('fda214a3-a8ed-40cc-bf70-60a807475c50'), 'worker_num': 3, 'host': 'datanode02-bi-dev', 'ppid': 26047, 'port': 19377, 'tb_pid': 0, 'tb_port': 0}
  52. 2017-02-24 11:28:21,201 INFO (MainThread-26057) node: {'addr': '/tmp/pymp-4wlOfY/listener-6eigjl', 'task_index': 3, 'job_name': 'worker', 'authkey': UUID('cb7d84ce-8fd2-445d-a3d4-d43b7e8f9e8e'), 'worker_num': 4, 'host': 'datanode04-bi-dev', 'ppid': 2409, 'port': 13204, 'tb_pid': 0, 'tb_port': 0}
  53. 2017-02-24 11:28:21,209 INFO (MainThread-26057) Connected to TFSparkNode.mgr on datanode02-bi-dev, ppid=26047, state='running'
  54. 2017-02-24 11:28:21,388 INFO (MainThread-26057) Starting TensorFlow worker:2 on cluster node 3 on background thread
  55. 17/02/24 11:28:21 INFO PythonRunner: Times: total = 202, boot = -545, init = 557, finish = 190
  56. 17/02/24 11:28:21 INFO Executor: Finished task 4.0 in stage 1.0 (TID 9). 2078 bytes result sent to driver
  57. 2017-02-24 11:28:22,199 INFO (Thread-1-26057) 3: ======== worker:2 ========
  58. 2017-02-24 11:28:22,200 INFO (Thread-1-26057) 3: Cluster spec: {'ps': ['datanode01-bi-dev:47797'], 'worker': ['namenode01-bi-dev:59678', 'datanode03-bi-dev:27900', 'datanode02-bi-dev:19377', 'datanode04-bi-dev:13204']}
  59. 2017-02-24 11:28:22,200 INFO (Thread-1-26057) 3: Using CPU
  60. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
  61. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
  62. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
  63. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
  64. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
  65. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
  66. D0224 11:28:22.201954983 26117 env_linux.c:77] Warning: insecure environment read function 'getenv' used
  67. I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:200] Initialize GrpcChannelCache for job ps -> {0 -> datanode01-bi-dev:47797}
  68. I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:200] Initialize GrpcChannelCache for job worker -> {0 -> namenode01-bi-dev:59678, 1 -> datanode03-bi-dev:27900, 2 -> localhost:19377, 3 -> datanode04-bi-dev:13204}
  69. I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:221] Started server with target: grpc://localhost:19377
  70. tensorflow model path: hdfs://ipipip:8020/user/hadoop/mnist_model
  71. 17/02/24 11:28:26 INFO CoarseGrainedExecutorBackend: Got assigned task 11
  72. 17/02/24 11:28:26 INFO Executor: Running task 1.0 in stage 2.0 (TID 11)
  73. 17/02/24 11:28:26 INFO TorrentBroadcast: Started reading broadcast variable 4
  74. 17/02/24 11:28:26 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 6.8 KB, free 912.3 MB)
  75. 17/02/24 11:28:26 INFO TorrentBroadcast: Reading broadcast variable 4 took 17 ms
  76. 17/02/24 11:28:26 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 12.8 KB, free 912.3 MB)
  77. 17/02/24 11:28:26 INFO HadoopRDD: Input split: hdfs://ipipip:8020/user/hadoop/mnist/csv/train/images/part-00001:0+11231804
  78. 17/02/24 11:28:26 INFO TorrentBroadcast: Started reading broadcast variable 0
  79. 17/02/24 11:28:26 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 21.1 KB, free 912.2 MB)
  80. 17/02/24 11:28:26 INFO TorrentBroadcast: Reading broadcast variable 0 took 17 ms
  81. 17/02/24 11:28:26 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 318.0 KB, free 911.9 MB)
  82. 17/02/24 11:28:26 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
  83. 17/02/24 11:28:26 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
  84. 17/02/24 11:28:26 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
  85. 17/02/24 11:28:26 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
  86. 17/02/24 11:28:26 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
  87. 17/02/24 11:28:27 INFO HadoopRDD: Input split: hdfs://ipipip:8020/user/hadoop/mnist/csv/train/labels/part-00001:0+245760
  88. 17/02/24 11:28:27 INFO TorrentBroadcast: Started reading broadcast variable 1
  89. 17/02/24 11:28:27 INFO TransportClientFactory: Successfully created connection to datanode04-bi-dev/10.104.90.44:31956 after 2 ms (0 ms spent in bootstraps)
  90. 17/02/24 11:28:27 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 21.1 KB, free 911.9 MB)
  91. 17/02/24 11:28:27 INFO TorrentBroadcast: Reading broadcast variable 1 took 25 ms
  92. 17/02/24 11:28:27 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 318.0 KB, free 911.6 MB)
  93. 2017-02-24 11:28:27,597 INFO (MainThread-26153) Connected to TFSparkNode.mgr on datanode02-bi-dev, ppid=26047, state='running'
  94. 2017-02-24 11:28:27,603 INFO (MainThread-26153) mgr.state='running'
  95. 2017-02-24 11:28:27,603 INFO (MainThread-26153) Feeding partition <generator object load_stream at 0x7f2eff36d5f0> into input queue <multiprocessing.queues.JoinableQueue object at 0x7f2ede8a6610>
  96. I tensorflow/core/distributed_runtime/master_session.cc:1012] Start master session 131488daafe1ccb2 with config:
  97.  
  98. INFO:tensorflow:Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad
  99. 2017-02-24 11:28:28,679 INFO (Thread-1-26057) Waiting for model to be ready. Ready_for_local_init_op: None, ready: Variables not initialized: hid_w, hid_b, sm_w, sm_b, Variable, hid_w/Adagrad, hid_b/Adagrad, sm_w/Adagrad, sm_b/Adagrad
  100. 17/02/24 11:28:30 INFO PythonRunner: Times: total = 3102, boot = -5628, init = 5695, finish = 3035
  101. 17/02/24 11:28:30 INFO PythonRunner: Times: total = 196, boot = 4, init = 61, finish = 131
  102. I tensorflow/core/distributed_runtime/master_session.cc:1012] Start master session 556be8bf537bf892 with config:
  103.  
  104. 2017-02-24T11:28:58.757118 session ready
  105. 2017-02-24T11:28:59.851874 step: 100 accuracy: 0.949999988079
  106. 2017-02-24 11:29:01,953 INFO (MainThread-26153) Processed 6144 items in partition
  107. 17/02/24 11:29:01 INFO PythonRunner: Times: total = 34404, boot = 3, init = 35, finish = 34366
  108. 17/02/24 11:29:01 INFO Executor: Finished task 1.0 in stage 2.0 (TID 11). 2442 bytes result sent to driver
  109. 17/02/24 11:29:01 INFO CoarseGrainedExecutorBackend: Got assigned task 16
  110. 17/02/24 11:29:01 INFO Executor: Running task 5.0 in stage 2.0 (TID 16)
  111. 17/02/24 11:29:01 INFO HadoopRDD: Input split: hdfs://ipipip:8020/user/hadoop/mnist/csv/train/images/part-00005:0+11173834
  112. 17/02/24 11:29:02 INFO HadoopRDD: Input split: hdfs://ipipip:8020/user/hadoop/mnist/csv/train/labels/part-00005:0+245760
  113. 2017-02-24 11:29:02,198 INFO (MainThread-26153) Connected to TFSparkNode.mgr on datanode02-bi-dev, ppid=26047, state='running'
  114. 2017-02-24 11:29:02,205 INFO (MainThread-26153) mgr.state='running'
  115. 2017-02-24 11:29:02,205 INFO (MainThread-26153) Feeding partition <generator object load_stream at 0x7f2ede8ac460> into input queue <multiprocessing.queues.JoinableQueue object at 0x7f2ede8a6610>
  116. 17/02/24 11:29:06 INFO PythonRunner: Times: total = 4227, boot = -31746, init = 31757, finish = 4216
  117. 17/02/24 11:29:06 INFO PythonRunner: Times: total = 60, boot = -34466, init = 34474, finish = 52
  118. 2017-02-24 11:29:09,202 INFO (MainThread-26153) Processed 6144 items in partition
  119. 17/02/24 11:29:09 INFO PythonRunner: Times: total = 7042, boot = -189, init = 220, finish = 7011
  120. 17/02/24 11:29:09 INFO Executor: Finished task 5.0 in stage 2.0 (TID 16). 1542 bytes result sent to driver
  121. 17/02/24 11:29:09 INFO CoarseGrainedExecutorBackend: Got assigned task 19
  122. 17/02/24 11:29:09 INFO Executor: Running task 9.0 in stage 2.0 (TID 19)
  123. 17/02/24 11:29:09 INFO HadoopRDD: Input split: hdfs://ipipip:8020/user/hadoop/mnist/csv/train/images/part-00009:0+10449019
  124. 17/02/24 11:29:09 INFO HadoopRDD: Input split: hdfs://ipipip:8020/user/hadoop/mnist/csv/train/labels/part-00009:0+229120
  125. 2017-02-24 11:29:09,425 INFO (MainThread-26153) Connected to TFSparkNode.mgr on datanode02-bi-dev, ppid=26047, state='running'
  126. 2017-02-24 11:29:09,433 INFO (MainThread-26153) mgr.state='running'
  127. 2017-02-24 11:29:09,433 INFO (MainThread-26153) Feeding partition <generator object load_stream at 0x7f2ede8ac5f0> into input queue <multiprocessing.queues.JoinableQueue object at 0x7f2ede8a6610>
  128. 17/02/24 11:29:13 INFO PythonRunner: Times: total = 4300, boot = -2985, init = 2994, finish = 4291
  129. 17/02/24 11:29:13 INFO PythonRunner: Times: total = 65, boot = -7157, init = 7165, finish = 57
  130. 2017-02-24 11:29:16,398 INFO (MainThread-26153) Processed 5728 items in partition
  131. 17/02/24 11:29:16 INFO PythonRunner: Times: total = 7014, boot = -175, init = 209, finish = 6980
  132. 17/02/24 11:29:16 INFO Executor: Finished task 9.0 in stage 2.0 (TID 19). 1382 bytes result sent to driver
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!