API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Feb 23rd, 2017
108
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 11.72 KB | None | 0 0
raw download clone embed print report
  1. SLF4J: Class path contains multiple SLF4J bindings.
  2. SLF4J: Found binding in [jar:file:/data/tmp/nm-local-dir/usercache/hadoop/filecache/25/__spark_libs__6843186552468170225.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  3. SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  4. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  5. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  6. 17/02/24 11:28:05 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 25797@namenode01-bi-dev
  7. 17/02/24 11:28:05 INFO SignalUtils: Registered signal handler for TERM
  8. 17/02/24 11:28:05 INFO SignalUtils: Registered signal handler for HUP
  9. 17/02/24 11:28:05 INFO SignalUtils: Registered signal handler for INT
  10. 17/02/24 11:28:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  11. 17/02/24 11:28:05 INFO SecurityManager: Changing view acls to: hadoop
  12. 17/02/24 11:28:05 INFO SecurityManager: Changing modify acls to: hadoop
  13. 17/02/24 11:28:05 INFO SecurityManager: Changing view acls groups to:
  14. 17/02/24 11:28:05 INFO SecurityManager: Changing modify acls groups to:
  15. 17/02/24 11:28:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
  16. 17/02/24 11:28:06 INFO TransportClientFactory: Successfully created connection to /111.111.90.40:37395 after 97 ms (0 ms spent in bootstraps)
  17. 17/02/24 11:28:06 INFO SecurityManager: Changing view acls to: hadoop
  18. 17/02/24 11:28:06 INFO SecurityManager: Changing modify acls to: hadoop
  19. 17/02/24 11:28:06 INFO SecurityManager: Changing view acls groups to:
  20. 17/02/24 11:28:06 INFO SecurityManager: Changing modify acls groups to:
  21. 17/02/24 11:28:06 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
  22. 17/02/24 11:28:06 INFO TransportClientFactory: Successfully created connection to /111.111.90.40:37395 after 1 ms (0 ms spent in bootstraps)
  23. 17/02/24 11:28:06 INFO DiskBlockManager: Created local directory at /data/tmp/nm-local-dir/usercache/hadoop/appcache/application_1487905765621_0005/blockmgr-7816cba3-dd7d-4a3f-b4ef-b34198b6dfbd
  24. 17/02/24 11:28:06 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
  25. 17/02/24 11:28:06 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@111.111.90.40:37395
  26. 17/02/24 11:28:06 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
  27. 17/02/24 11:28:06 INFO Executor: Starting executor ID 4 on host namenode01-bi-dev
  28. 17/02/24 11:28:06 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 63881.
  29. 17/02/24 11:28:06 INFO NettyBlockTransferService: Server created on namenode01-bi-dev:63881
  30. 17/02/24 11:28:06 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(4, namenode01-bi-dev, 63881)
  31. 17/02/24 11:28:06 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(4, namenode01-bi-dev, 63881)
  32. 17/02/24 11:28:09 INFO CoarseGrainedExecutorBackend: Got assigned task 1
  33. 17/02/24 11:28:09 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
  34. 17/02/24 11:28:10 INFO TorrentBroadcast: Started reading broadcast variable 2
  35. 17/02/24 11:28:10 INFO TransportClientFactory: Successfully created connection to /111.111.90.40:34436 after 2 ms (0 ms spent in bootstraps)
  36. 17/02/24 11:28:10 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 4.2 KB, free 912.3 MB)
  37. 17/02/24 11:28:10 INFO TorrentBroadcast: Reading broadcast variable 2 took 118 ms
  38. 17/02/24 11:28:10 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 5.9 KB, free 912.3 MB)
  39. 2017-02-24 11:28:20,568 INFO (MainThread-25877) TFSparkNode.reserve: {'authkey': UUID('256af5f7-287a-4c4d-8165-9816bff21808'), 'worker_num': 1, 'host': 'namenode01-bi-dev', 'tb_port': 0, 'addr': '/tmp/pymp-jCE5VH/listener-zlzwwH', 'ppid': 25867, 'task_index': 0, 'job_name': 'worker', 'tb_pid': 0, 'port': 59678}
  40. 17/02/24 11:28:20 INFO PythonRunner: Times: total = 10308, boot = 236, init = 43, finish = 10029
  41. 17/02/24 11:28:20 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 2314 bytes result sent to driver
  42. 17/02/24 11:28:21 INFO CoarseGrainedExecutorBackend: Got assigned task 8
  43. 17/02/24 11:28:21 INFO Executor: Running task 3.0 in stage 1.0 (TID 8)
  44. 17/02/24 11:28:21 INFO TorrentBroadcast: Started reading broadcast variable 3
  45. 17/02/24 11:28:21 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 6.3 KB, free 912.3 MB)
  46. 17/02/24 11:28:21 INFO TorrentBroadcast: Reading broadcast variable 3 took 19 ms
  47. 17/02/24 11:28:21 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 9.3 KB, free 912.3 MB)
  48. 2017-02-24 11:28:21,209 INFO (MainThread-25877) node: {'addr': ('datanode01-bi-dev', 12241), 'task_index': 0, 'job_name': 'ps', 'authkey': UUID('0843b53f-22a3-4cf5-ab0d-d316ba7f792c'), 'worker_num': 0, 'host': 'datanode01-bi-dev', 'ppid': 28724, 'port': 47797, 'tb_pid': 0, 'tb_port': 0}
  49. 2017-02-24 11:28:21,209 INFO (MainThread-25877) node: {'addr': '/tmp/pymp-jCE5VH/listener-zlzwwH', 'task_index': 0, 'job_name': 'worker', 'authkey': UUID('256af5f7-287a-4c4d-8165-9816bff21808'), 'worker_num': 1, 'host': 'namenode01-bi-dev', 'ppid': 25867, 'port': 59678, 'tb_pid': 0, 'tb_port': 0}
  50. 2017-02-24 11:28:21,209 INFO (MainThread-25877) node: {'addr': '/tmp/pymp-vMwVR6/listener-w9GOmI', 'task_index': 1, 'job_name': 'worker', 'authkey': UUID('1b726e7a-c176-4974-8cb7-a3984d54028f'), 'worker_num': 2, 'host': 'datanode03-bi-dev', 'ppid': 24048, 'port': 27900, 'tb_pid': 0, 'tb_port': 0}
  51. 2017-02-24 11:28:21,209 INFO (MainThread-25877) node: {'addr': '/tmp/pymp-WZymp4/listener-ilxHlD', 'task_index': 2, 'job_name': 'worker', 'authkey': UUID('fda214a3-a8ed-40cc-bf70-60a807475c50'), 'worker_num': 3, 'host': 'datanode02-bi-dev', 'ppid': 26047, 'port': 19377, 'tb_pid': 0, 'tb_port': 0}
  52. 2017-02-24 11:28:21,210 INFO (MainThread-25877) node: {'addr': '/tmp/pymp-4wlOfY/listener-6eigjl', 'task_index': 3, 'job_name': 'worker', 'authkey': UUID('cb7d84ce-8fd2-445d-a3d4-d43b7e8f9e8e'), 'worker_num': 4, 'host': 'datanode04-bi-dev', 'ppid': 2409, 'port': 13204, 'tb_pid': 0, 'tb_port': 0}
  53. 2017-02-24 11:28:21,217 INFO (MainThread-25877) Connected to TFSparkNode.mgr on namenode01-bi-dev, ppid=25867, state='running'
  54. 2017-02-24 11:28:21,377 INFO (MainThread-25877) Starting TensorFlow worker:0 on cluster node 1 on background thread
  55. 17/02/24 11:28:21 INFO PythonRunner: Times: total = 183, boot = -609, init = 622, finish = 170
  56. 17/02/24 11:28:21 INFO Executor: Finished task 3.0 in stage 1.0 (TID 8). 2078 bytes result sent to driver
  57. 2017-02-24 11:28:22,220 INFO (Thread-1-25877) 1: ======== worker:0 ========
  58. 2017-02-24 11:28:22,221 INFO (Thread-1-25877) 1: Cluster spec: {'ps': ['datanode01-bi-dev:47797'], 'worker': ['namenode01-bi-dev:59678', 'datanode03-bi-dev:27900', 'datanode02-bi-dev:19377', 'datanode04-bi-dev:13204']}
  59. 2017-02-24 11:28:22,221 INFO (Thread-1-25877) 1: Using CPU
  60. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
  61. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
  62. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
  63. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
  64. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
  65. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
  66. D0224 11:28:22.223334300 25939 env_linux.c:77] Warning: insecure environment read function 'getenv' used
  67. I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:200] Initialize GrpcChannelCache for job ps -> {0 -> datanode01-bi-dev:47797}
  68. I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:200] Initialize GrpcChannelCache for job worker -> {0 -> localhost:59678, 1 -> datanode03-bi-dev:27900, 2 -> datanode02-bi-dev:19377, 3 -> datanode04-bi-dev:13204}
  69. I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:221] Started server with target: grpc://localhost:59678
  70. tensorflow model path: hdfs://111.111.90.40:8020/user/hadoop/mnist_model
  71. 17/02/24 11:28:29 INFO CoarseGrainedExecutorBackend: Got assigned task 13
  72. 17/02/24 11:28:29 INFO Executor: Running task 3.0 in stage 2.0 (TID 13)
  73. 17/02/24 11:28:29 INFO TorrentBroadcast: Started reading broadcast variable 4
  74. 17/02/24 11:28:29 INFO TransportClientFactory: Successfully created connection to datanode03-bi-dev/111.111.90.43:60029 after 2 ms (0 ms spent in bootstraps)
  75. 17/02/24 11:28:29 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 6.8 KB, free 912.3 MB)
  76. 17/02/24 11:28:29 INFO TorrentBroadcast: Reading broadcast variable 4 took 57 ms
  77. 17/02/24 11:28:29 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 12.8 KB, free 912.3 MB)
  78. 17/02/24 11:28:29 INFO HadoopRDD: Input split: hdfs://111.111.90.40:8020/user/hadoop/mnist/csv/train/images/part-00003:0+11226100
  79. 17/02/24 11:28:29 INFO TorrentBroadcast: Started reading broadcast variable 0
  80. 17/02/24 11:28:29 INFO TransportClientFactory: Successfully created connection to datanode02-bi-dev/111.111.90.42:8868 after 2 ms (0 ms spent in bootstraps)
  81. 17/02/24 11:28:29 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 21.1 KB, free 912.2 MB)
  82. 17/02/24 11:28:29 INFO odcast: Reading broadcast variable 0 took 62 ms
  83. 17/02/24 11:28:29 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 318.0 KB, free 911.9 MB)
  84. 17/02/24 11:28:30 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
  85. 17/02/24 11:28:30 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
  86. 17/02/24 11:28:30 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
  87. 17/02/24 11:28:30 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
  88. 17/02/24 11:28:30 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
  89. 17/02/24 11:28:30 INFO HadoopRDD: Input split: hdfs://111.111.90.40:8020/user/hadoop/mnist/csv/train/labels/part-00003:0+245760
  90. 17/02/24 11:28:30 INFO TorrentBroadcast: Started reading broadcast variable 1
  91. 17/02/24 11:28:30 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 21.1 KB, free 911.9 MB)
  92. 17/02/2nput queue <multiprocessing.queues.JoinableQueue object at 0x7ff6d0bfb210>
  93. I tensorflow/core/distributed_runtime/master_session.cc:1012] Start master session 5e35d3174c4ef0b0 with config:
  94.  
  95. 17/02/24 11:28:33 INFO PythonRunner: Times: total = 3183, boot = -9015, init = 9085, finish = 3113
  96. 17/02/24 11:28:33 INFO PythonRunner: Times: total = 214, boot = 3, init = 62, finish = 149orren
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!