API tools faq
paste
Guest User

Untitled

a guest
Jan 23rd, 2019
98
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 20.12 KB | None | 0 0
raw download clone embed print report
  1. INFO:tensorflow:Done calling model_fn.
  2. INFO:tensorflow:Create CheckpointSaverHook.
  3. 2019-01-18 15:37:45.270305: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404
  4. 2019-01-18 15:37:45.270348: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  5. 2019-01-18 15:37:45.286015: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404
  6. 2019-01-18 15:37:45.286048: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  7. 2019-01-18 15:37:45.302541: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404
  8. 2019-01-18 15:37:45.302580: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  9. INFO:tensorflow:Graph was finalized.
  10. 2019-01-18 15:37:48.674525: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404
  11. 2019-01-18 15:37:48.674567: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  12. INFO:tensorflow:Running local_init_op.
  13. INFO:tensorflow:Done running local_init_op.
  14. 2019-01-18 15:37:52.494279: E tensorflow/core/platform/s3/aws_logging.cc:60] No response body. Response code: 404
  15. 2019-01-18 15:37:52.494323: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  16. 2019-01-18 15:38:09.905949: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
  17. 2019-01-18 15:38:09.906012: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  18. 2019-01-18 15:38:09.906026: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 0 ms before attempting again.
  19. 2019-01-18 15:38:13.910953: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
  20. 2019-01-18 15:38:13.911018: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  21. 2019-01-18 15:38:13.911032: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 50 ms before attempting again.
  22. 2019-01-18 15:38:16.993509: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
  23. 2019-01-18 15:38:16.993915: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  24. 2019-01-18 15:38:16.993944: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 100 ms before attempting again.
  25. 2019-01-18 15:38:21.098749: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
  26. 2019-01-18 15:38:21.098814: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  27. 2019-01-18 15:38:21.098828: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 200 ms before attempting again.
  28. 2019-01-18 15:38:24.504661: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
  29. 2019-01-18 15:38:24.504723: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  30. 2019-01-18 15:38:24.504741: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 400 ms before attempting again.
  31. 2019-01-18 15:38:28.266722: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
  32. 2019-01-18 15:38:28.266779: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  33. 2019-01-18 15:38:28.266798: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 800 ms before attempting again.
  34. INFO:tensorflow:Saving checkpoints for 0 into s3://bucket/2019Jan18/1533/model.ckpt.
  35. 2019-01-18 15:38:54.233802: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
  36. 2019-01-18 15:38:54.233869: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  37. 2019-01-18 15:38:54.233888: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 0 ms before attempting again.
  38. 2019-01-18 15:38:58.238623: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
  39. 2019-01-18 15:38:58.238692: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  40. 2019-01-18 15:38:58.238712: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 50 ms before attempting again.
  41. 2019-01-18 15:39:02.293232: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
  42. 2019-01-18 15:39:02.293295: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  43. 2019-01-18 15:39:02.293314: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 100 ms before attempting again.
  44. 2019-01-18 15:39:06.399724: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
  45. 2019-01-18 15:39:06.399791: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  46. 2019-01-18 15:39:06.399812: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 200 ms before attempting again.
  47. 2019-01-18 15:39:10.605093: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
  48. 2019-01-18 15:39:10.605155: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  49. 2019-01-18 15:39:10.605174: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 400 ms before attempting again.
  50. 2019-01-18 15:39:15.011578: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
  51. 2019-01-18 15:39:15.011647: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  52. 2019-01-18 15:39:15.011669: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 800 ms before attempting again.
  53. 2019-01-18 15:39:19.818554: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
  54. 2019-01-18 15:39:19.818621: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  55. 2019-01-18 15:39:19.818641: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 1600 ms before attempting again.
  56. 2019-01-18 15:39:25.423964: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
  57. 2019-01-18 15:39:25.424020: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  58. 2019-01-18 15:39:25.424045: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 3200 ms before attempting again.
  59. 2019-01-18 15:39:32.631174: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
  60. 2019-01-18 15:39:32.631239: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  61. 2019-01-18 15:39:32.631261: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 6400 ms before attempting again.
  62. 2019-01-18 15:39:43.037085: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
  63. 2019-01-18 15:39:43.037142: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  64. 2019-01-18 15:39:43.037161: W tensorflow/core/platform/s3/aws_logging.cc:57] Request failed, now waiting 12800 ms before attempting again.
  65.  
  66. 2019-01-18 15:40:10 Uploading - Uploading generated training model
  67. 2019-01-18 15:40:10 Failed - Training job failed
  68. 2019-01-18 15:39:59.843792: E tensorflow/core/platform/s3/aws_logging.cc:60] Curl returned error code 28
  69. 2019-01-18 15:39:59.843850: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
  70. 2019-01-18 15:39:59.843898: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at save_restore_v2_ops.cc:137 : Unknown: : Unable to connect to endpoint
  71. Traceback (most recent call last):
  72. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1334, in _do_call
  73. return fn(*args)
  74. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
  75. options, feed_dict, fetch_list, target_list, run_metadata)
  76. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
  77. run_metadata)
  78. tensorflow.python.framework.errors_impl.UnknownError: : Unable to connect to endpoint
  79. #011 [[{{node save/SaveV2_1}} = SaveV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](save/ShardedFilename_1, save/SaveV2_1/tensor_names, save/SaveV2_1/shape_and_slices, {other tensors}
  80.  
  81. During handling of the above exception, another exception occurred:
  82.  
  83. Traceback (most recent call last):
  84. File "estimator/estimator_main.py", line 304, in <module>
  85. main()
  86. File "estimator/estimator_main.py", line 204, in main
  87. eval_spec=eval_spec
  88. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 471, in train_and_evaluate
  89. return executor.run()
  90. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 610, in run
  91. return self.run_local()
  92. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 711, in run_local
  93. saving_listeners=saving_listeners)
  94. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 354, in train
  95. loss = self._train_model(input_fn, hooks, saving_listeners)
  96. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1207, in _train_model
  97. return self._train_model_default(input_fn, hooks, saving_listeners)
  98. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1241, in _train_model_default
  99. saving_listeners)
  100. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1468, in _train_with_estimator_spec
  101. log_step_count_steps=log_step_count_steps) as mon_sess:
  102. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 504, in MonitoredTrainingSession
  103. stop_grace_period_secs=stop_grace_period_secs)
  104. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 921, in __init__
  105. stop_grace_period_secs=stop_grace_period_secs)
  106. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 643, in __init__
  107. self._sess = _RecoverableSession(self._coordinated_creator)
  108. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1107, in __init__
  109. _WrappedSession.__init__(self, self._create_session())
  110. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1112, in _create_session
  111. return self._sess_creator.create_session()
  112. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 807, in create_session
  113. hook.after_create_session(self.tf_sess, self.coord)
  114. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/basic_session_run_hooks.py", line 568, in after_create_session
  115. self._save(session, global_step)
  116. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/basic_session_run_hooks.py", line 599, in _save
  117. self._get_saver().save(session, self._save_path, global_step=step)
  118. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1441, in save
  119. {self.saver_def.filename_tensor_name: checkpoint_file})
  120. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 929, in run
  121. run_metadata_ptr)
  122. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1152, in _run
  123. feed_dict_tensor, options, run_metadata)
  124. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run
  125. run_metadata)
  126. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call
  127. raise type(e)(node_def, op, message)
  128. tensorflow.python.framework.errors_impl.UnknownError: : Unable to connect to endpoint
  129. #011 [[node save/SaveV2_1 (defined at estimator/estimator_main.py:204) = SaveV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](save/ShardedFilename_1, save/SaveV2_1/tensor_names, save/SaveV2_1/shape_and_slices, apply_gradients/beta1_power/_237, apply_gradients/beta2_power/_239, model/item_embeddings/_241, model/item_embeddings/Adam/_243, model/item_embeddings/Adam_1/_245, model/output_b/_247, model/output_b/Adam/_249, model/output_b/Adam_1/_251, model/output_w/_253, model/output_w/Adam/_255, model/output_w/Adam_1/_257, model/rnn/time_delta_cell/h_gate_bias/_259, model/rnn/time_delta_cell/h_gate_bias/Adam/_261, model/rnn/time_delta_cell/h_gate_bias/Adam_1/_263, model/rnn/time_delta_cell/h_gate_weight/_265, model/rnn/time_delta_cell/h_gate_weight/Adam/_267, model/rnn/time_delta_cell/h_gate_weight/Adam_1/_269, model/rnn/time_delta_cell/update_vector_h_bias/_271, model/rnn/time_delta_cell/update_vector_h_bias/Adam/_273, model/rnn/time_delta_cell/update_vector_h_bias/Adam_1/_275, model/rnn/time_delta_cell/update_vector_h_weight/_277, model/rnn/time_delta_cell/update_vector_h_weight/Adam/_279, model/rnn/time_delta_cell/update_vector_h_weight/Adam_1/_281, model/rnn/time_delta_cell/update_vector_i_bias/_283, model/rnn/time_delta_cell/update_vector_i_bias/Adam/_285, model/rnn/time_delta_cell/update_vector_i_bias/Adam_1/_287, model/rnn/time_delta_cell/update_vector_i_weight/_289, model/rnn/time_delta_cell/update_vector_i_weight/Adam/_291, model/rnn/time_delta_cell/update_vector_i_weight/Adam_1/_293, model/rnn/time_delta_cell/update_vector_u_bias/_295, model/rnn/time_delta_cell/update_vector_u_bias/Adam/_297, model/rnn/time_delta_cell/update_vector_u_bias/Adam_1/_299, model/rnn/time_delta_cell/update_vector_u_weight/_301, model/rnn/time_delta_cell/update_vector_u_weight/Adam/_303, model/rnn/time_delta_cell/update_vector_u_weight/Adam_1/_305, model/rnn/time_delta_
  130.  
  131. Caused by op 'save/SaveV2_1', defined at:
  132. File "estimator/estimator_main.py", line 304, in <module>
  133. main()
  134. File "estimator/estimator_main.py", line 204, in main
  135. eval_spec=eval_spec
  136. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 471, in train_and_evaluate
  137. return executor.run()
  138. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 610, in run
  139. return self.run_local()
  140. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 711, in run_local
  141. saving_listeners=saving_listeners)
  142. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 354, in train
  143. loss = self._train_model(input_fn, hooks, saving_listeners)
  144. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1207, in _train_model
  145. return self._train_model_default(input_fn, hooks, saving_listeners)
  146. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1241, in _train_model_default
  147. saving_listeners)
  148. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1468, in _train_with_estimator_spec
  149. log_step_count_steps=log_step_count_steps) as mon_sess:
  150. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 504, in MonitoredTrainingSession
  151. stop_grace_period_secs=stop_grace_period_secs)
  152. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 921, in __init__
  153. stop_grace_period_secs=stop_grace_period_secs)
  154. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 643, in __init__
  155. self._sess = _RecoverableSession(self._coordinated_creator)
  156. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1107, in __init__
  157. _WrappedSession.__init__(self, self._create_session())
  158. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1112, in _create_session
  159. return self._sess_creator.create_session()
  160. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 800, in create_session
  161. self.tf_sess = self._session_creator.create_session()
  162. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 557, in create_session
  163. self._scaffold.finalize()
  164. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 215, in finalize
  165. self._saver.build()
  166. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1114, in build
  167. self._build(self._filename, build_save=True, build_restore=True)
  168. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1151, in _build
  169. build_save=build_save, build_restore=build_restore)
  170. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 786, in _build_internal
  171. save_tensor = self._AddShardedSaveOps(filename_tensor, per_device)
  172. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 369, in _AddShardedSaveOps
  173. return self._AddShardedSaveOpsForV2(filename_tensor, per_device)
  174. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 343, in _AddShardedSaveOpsForV2
  175. sharded_saves.append(self._AddSaveOps(sharded_filename, saveables))
  176. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 284, in _AddSaveOps
  177. save = self.save_op(filename_tensor, saveables)
  178. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 202, in save_op
  179. tensors)
  180. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 1690, in save_v2
  181. shape_and_slices=shape_and_slices, tensors=tensors, name=name)
  182. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
  183. op_def=op_def)
  184. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
  185. return func(*args, **kwargs)
  186. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
  187. op_def=op_def)
  188. File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
  189. self._traceback = tf_stack.extract_stack()
  190.  
  191. UnknownError (see above for traceback): : Unable to connect to endpoint
  192. #011 [[node save/SaveV2_1 (defined at estimator/estimator_main.py:204) = SaveV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](save/ShardedFilename_1, save/SaveV2_1/tensor_names, save/SaveV2_1/shape_and_slices, {other tensors}
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!