API tools faq
paste
Guest User

Untitled

a guest
Jan 23rd, 2019
103
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 19.20 KB | None | 0 0
raw download clone embed print report
  1. NFO:tensorflow:Average examples/sec: 1374.58 (7303.88), step = 470
  2. 2019-01-21 13:06:37.354097: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404
  3. 2019-01-21 13:06:37.354132: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  4. 2019-01-21 13:06:37.491716: W tensorflow/core/platform/s3/aws_logging.cc:57] Encountered Unknown AWSError
  5. InvalidRange
  6. The requested range is not satisfiable:
  7. 2019-01-21 13:06:37.491757: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  8. 2019-01-21 13:06:37.680168: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404
  9. 2019-01-21 13:06:37.680202: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  10. 2019-01-21 13:06:37.765341: W tensorflow/core/platform/s3/aws_logging.cc:57] Encountered Unknown AWSError
  11. InvalidRange
  12. The requested range is not satisfiable:
  13. 2019-01-21 13:06:37.765388: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  14. 2019-01-21 13:07:38.925475: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404
  15. 2019-01-21 13:07:38.925523: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  16. 2019-01-21 13:07:39.080818: W tensorflow/core/platform/s3/aws_logging.cc:57] Encountered Unknown AWSError
  17. InvalidRange
  18. The requested range is not satisfiable:
  19. 2019-01-21 13:07:39.080867: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  20. 2019-01-21 13:07:39.267329: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404
  21. 2019-01-21 13:07:39.267372: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  22. 2019-01-21 13:07:39.345499: W tensorflow/core/platform/s3/aws_logging.cc:57] Encountered Unknown AWSError
  23. InvalidRange
  24. The requested range is not satisfiable:
  25. 2019-01-21 13:07:39.345539: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  26. 2019-01-21 13:07:39.536645: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404
  27. 2019-01-21 13:07:39.536701: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  28. 2019-01-21 13:07:39.619464: W tensorflow/core/platform/s3/aws_logging.cc:57] Encountered Unknown AWSError
  29. InvalidRange
  30. The requested range is not satisfiable:
  31. 2019-01-21 13:07:39.619504: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  32. 2019-01-21 13:07:39.804496: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404
  33. 2019-01-21 13:07:39.804533: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  34. 2019-01-21 13:07:39.888266: W tensorflow/core/platform/s3/aws_logging.cc:57] Encountered Unknown AWSError
  35. InvalidRange
  36. The requested range is not satisfiable:
  37. 2019-01-21 13:07:39.888305: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  38. 2019-01-21 13:07:40.080018: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404
  39. 2019-01-21 13:07:40.080070: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  40. 2019-01-21 13:07:40.166409: W tensorflow/core/platform/s3/aws_logging.cc:57] Encountered Unknown AWSError
  41. InvalidRange
  42. The requested range is not satisfiable:
  43. 2019-01-21 13:07:40.166455: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  44. 2019-01-21 13:07:40.356976: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404
  45. 2019-01-21 13:07:40.357015: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  46. 2019-01-21 13:07:40.449878: W tensorflow/core/platform/s3/aws_logging.cc:57] Encountered Unknown AWSError
  47. InvalidRange
  48. The requested range is not satisfiable:
  49. 2019-01-21 13:07:40.449928: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  50. 2019-01-21 13:07:40.639521: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404
  51. 2019-01-21 13:07:40.639570: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  52. 2019-01-21 13:07:40.729489: W tensorflow/core/platform/s3/aws_logging.cc:57] Encountered Unknown AWSError
  53. InvalidRange
  54. The requested range is not satisfiable:
  55. 2019-01-21 13:07:40.729538: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  56. INFO:tensorflow:Saving checkpoints for 480 into s3://bucket/2019Jan21/1249/model.ckpt.
  57. 2019-01-21 13:07:41.046769: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 55
  58. 2019-01-21 13:07:41.046847: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  59. 2019-01-21 13:07:41.046867: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 0 ms before attempting again.
  60. 2019-01-21 13:07:41.135201: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 55
  61. 2019-01-21 13:07:41.135566: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  62. 2019-01-21 13:07:41.135966: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 50 ms before attempting again.
  63. 2019-01-21 13:07:41.250242: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 55
  64. 2019-01-21 13:07:41.250328: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  65. 2019-01-21 13:07:41.251001: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 100 ms before attempting again.
  66. 2019-01-21 13:07:41.398659: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 55
  67. 2019-01-21 13:07:41.398732: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  68. 2019-01-21 13:07:41.399407: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 200 ms before attempting again.
  69. 2019-01-21 13:07:41.618052: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 55
  70. 2019-01-21 13:07:41.618131: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  71. 2019-01-21 13:07:41.618152: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 400 ms before attempting again.
  72. 2019-01-21 13:07:42.097115: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 55
  73. 2019-01-21 13:07:42.097441: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  74. 2019-01-21 13:07:42.097839: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 800 ms before attempting again.
  75. 2019-01-21 13:07:42.975713: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 55
  76. 2019-01-21 13:07:42.975770: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  77. 2019-01-21 13:07:42.976433: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 1600 ms before attempting again.
  78. 2019-01-21 13:07:44.658061: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 55
  79. 2019-01-21 13:07:44.658115: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  80. 2019-01-21 13:07:44.658769: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 3200 ms before attempting again.
  81. 2019-01-21 13:07:47.935585: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 55
  82. 2019-01-21 13:07:47.935643: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  83. 2019-01-21 13:07:47.936298: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 6400 ms before attempting again.
  84. 2019-01-21 13:07:54.392356: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 55
  85. 2019-01-21 13:07:54.392412: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  86. 2019-01-21 13:07:54.393061: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 12800 ms before attempting again.
  87. 2019-01-21 13:08:07.251103: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 55
  88. 2019-01-21 13:08:07.251418: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  89. 2019-01-21 13:08:07.340006: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at save_restore_v2_ops.cc:137 : Unknown: : Unable to connect to endpoint
  90. Traceback (most recent call last):
  91. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1334, in _do_call
  92. return fn(*args)
  93. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
  94. options, feed_dict, fetch_list, target_list, run_metadata)
  95. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
  96. run_metadata)
  97. tensorflow.python.framework.errors_impl.UnknownError: : Unable to connect to endpoint
  98. #011 [[{{node save/SaveV2}} = SaveV2[dtypes=[DT_INT64, DT_BOOL], _device="/job:localhost/replica:0/task:0/device:CPU:0"](save/ShardedFilename, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, global_step/_283, signal_early_stopping/STOP)]]
  99. During handling of the above exception, another exception occurred:
  100. Traceback (most recent call last):
  101. File "estimator/estimator_main.py", line 310, in <module>
  102. main()
  103. File "estimator/estimator_main.py", line 210, in main
  104. eval_spec=eval_spec
  105. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 471, in train_and_evaluate
  106. return executor.run()
  107. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 610, in run
  108. return self.run_local()
  109. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 711, in run_local
  110. saving_listeners=saving_listeners)
  111. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 354, in train
  112. loss = self._train_model(input_fn, hooks, saving_listeners)
  113. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1207, in _train_model
  114. return self._train_model_default(input_fn, hooks, saving_listeners)
  115. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1241, in _train_model_default
  116. saving_listeners)
  117. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1471, in _train_with_estimator_spec
  118. _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  119. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 671, in run
  120. run_metadata=run_metadata)
  121. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1156, in run
  122. run_metadata=run_metadata)
  123. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1255, in run
  124. raise six.reraise(*original_exc_info)
  125. File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
  126. raise value
  127. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1240, in run
  128. return self._sess.run(*args, **kwargs)
  129. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1320, in run
  130. run_metadata=run_metadata))
  131. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/basic_session_run_hooks.py", line 582, in after_run
  132. if self._save(run_context.session, global_step):
  133. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/basic_session_run_hooks.py", line 599, in _save
  134. self._get_saver().save(session, self._save_path, global_step=step)
  135. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1441, in save
  136. {self.saver_def.filename_tensor_name: checkpoint_file})
  137. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 929, in run
  138. run_metadata_ptr)
  139. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1152, in _run
  140. feed_dict_tensor, options, run_metadata)
  141. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run
  142. run_metadata)
  143. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call
  144. raise type(e)(node_def, op, message)
  145. tensorflow.python.framework.errors_impl.UnknownError: : Unable to connect to endpoint
  146. #011 [[node save/SaveV2 (defined at estimator/estimator_main.py:210) = SaveV2[dtypes=[DT_INT64, DT_BOOL], _device="/job:localhost/replica:0/task:0/device:CPU:0"](save/ShardedFilename, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, global_step/_283, signal_early_stopping/STOP)]]
  147. Caused by op 'save/SaveV2', defined at:
  148. File "estimator/estimator_main.py", line 310, in <module>
  149. main()
  150. File "estimator/estimator_main.py", line 210, in main
  151. eval_spec=eval_spec
  152. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 471, in train_and_evaluate
  153. return executor.run()
  154. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 610, in run
  155. return self.run_local()
  156. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 711, in run_local
  157. saving_listeners=saving_listeners)
  158. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 354, in train
  159. loss = self._train_model(input_fn, hooks, saving_listeners)
  160. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1207, in _train_model
  161. return self._train_model_default(input_fn, hooks, saving_listeners)
  162. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1241, in _train_model_default
  163. saving_listeners)
  164. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1468, in _train_with_estimator_spec
  165. log_step_count_steps=log_step_count_steps) as mon_sess:
  166. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 504, in MonitoredTrainingSession
  167. stop_grace_period_secs=stop_grace_period_secs)
  168. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 921, in __init__
  169. stop_grace_period_secs=stop_grace_period_secs)
  170. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 643, in __init__
  171. self._sess = _RecoverableSession(self._coordinated_creator)
  172. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1107, in __init__
  173. _WrappedSession.__init__(self, self._create_session())
  174. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1112, in _create_session
  175. return self._sess_creator.create_session()
  176. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 800, in create_session
  177. self.tf_sess = self._session_creator.create_session()
  178. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 557, in create_session
  179. self._scaffold.finalize()
  180. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 215, in finalize
  181. self._saver.build()
  182. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1114, in build
  183. self._build(self._filename, build_save=True, build_restore=True)
  184. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1151, in _build
  185. build_save=build_save, build_restore=build_restore)
  186. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 786, in _build_internal
  187. save_tensor = self._AddShardedSaveOps(filename_tensor, per_device)
  188. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 369, in _AddShardedSaveOps
  189. return self._AddShardedSaveOpsForV2(filename_tensor, per_device)
  190. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 343, in _AddShardedSaveOpsForV2
  191. sharded_saves.append(self._AddSaveOps(sharded_filename, saveables))
  192. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 284, in _AddSaveOps
  193. save = self.save_op(filename_tensor, saveables)
  194. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 202, in save_op
  195. tensors)
  196. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 1690, in save_v2
  197. shape_and_slices=shape_and_slices, tensors=tensors, name=name)
  198. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
  199. op_def=op_def)
  200. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
  201. return func(*args, **kwargs)
  202. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
  203. op_def=op_def)
  204. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
  205. self._traceback = tf_stack.extract_stack()
  206. UnknownError (see above for traceback): : Unable to connect to endpoint
  207. #011 [[node save/SaveV2 (defined at estimator/estimator_main.py:210) = SaveV2[dtypes=[DT_INT64, DT_BOOL], _device="/job:localhost/replica:0/task:0/device:CPU:0"](save/ShardedFilename, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, global_step/_283, signal_early_stopping/STOP)]]
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!