API tools faq
paste
Advertisement
nickwalton

training error

nickwalton
Oct 22nd, 2019
135
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Bash 15.17 KB | None | 0 0
raw download clone embed print report
  1. mon_runtime/placer.cc:54] Placeholder_759: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  2. Placeholder_760: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  3. 2019-10-22 19:25:05.058291: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_760: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  4. Placeholder_761: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  5. 2019-10-22 19:25:05.058313: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_761: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  6. Placeholder_762: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  7. 2019-10-22 19:25:05.058333: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_762: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  8. Placeholder_763: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  9. 2019-10-22 19:25:05.058354: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_763: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  10. Placeholder_764: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  11. 2019-10-22 19:25:05.058375: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_764: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  12. Placeholder_765: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  13. 2019-10-22 19:25:05.058395: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_765: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  14. Placeholder_766: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  15. 2019-10-22 19:25:05.058416: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_766: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  16. Placeholder_767: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  17. 2019-10-22 19:25:05.058455: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_767: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  18. Placeholder_768: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  19. 2019-10-22 19:25:05.058476: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_768: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  20. Placeholder_769: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  21. 2019-10-22 19:25:05.058497: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_769: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  22. Placeholder_770: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  23. 2019-10-22 19:25:05.058518: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_770: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  24. Placeholder_771: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  25. 2019-10-22 19:25:05.058539: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_771: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  26. 2019-10-22 19:25:08.834289: W tensorflow/core/framework/allocator.cc:107] Allocation of 1262254080 exceeds 10% of system memory.
  27. 2019-10-22 19:25:11.916920: W tensorflow/core/framework/allocator.cc:107] Allocation of 1262254080 exceeds 10% of system memory.
  28. W1022 19:25:38.527546 139874682042176 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/clip_ops.py:286: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
  29. Instructions for updating:
  30. Use tf.where in 2.0, which has the same broadcast rule as np.where
  31. W1022 19:25:39.403865 139874682042176 deprecation.py:506] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/adagrad.py:76: calling __init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
  32. Instructions for updating:
  33. Call initializer instance with the dtype argument instead of passing it to the constructor
  34. global_step: (VariableV2): /job:localhost/replica:0/task:0/device:GPU:0
  35. 2019-10-22 19:25:48.848774: I tensorflow/core/common_runtime/placer.cc:54] global_step: (VariableV2)/job:localhost/replica:0/task:0/device:GPU:0
  36. global_step/Assign: (Assign): /job:localhost/replica:0/task:0/device:GPU:0
  37. 2019-10-22 19:25:48.848855: I tensorflow/core/common_runtime/placer.cc:54] global_step/Assign: (Assign)/job:localhost/replica:0/task:0/device:GPU:0
  38. global_step/read: (Identity): /job:localhost/replica:0/task:0/device:GPU:0
  39. 2019-10-22 19:25:48.848868: I tensorflow/core/common_runtime/placer.cc:54] global_step/read: (Identity)/job:localhost/replica:0/task:0/device:GPU:0
  40. w/Initializer/random_normal/RandomStandardNormal: (RandomStandardNormal): /job:localhost/replica:0/task:0/device:GPU:0
  41. 2019-10-22 19:25:48.848897: I tensorflow/core/common_runtime/placer.cc:54] w/Initializer/random_normal/RandomStandardNormal: (RandomStandardNormal)/job:localhost/replica:0/task:0/device:GPU:0
  42. Traceback (most recent call last):
  43.   File "training.py", line 162, in <module>
  44.     estimator_model = tf.keras.estimator.model_to_estimator(keras_model=model, config=run_config)
  45.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/estimator/__init__.py", line 73, in model_to_estimator
  46.     config=config)
  47.   File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/keras.py", line 450, in model_to_estimator
  48.     config)
  49.   File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/keras.py", line 331, in _save_first_checkpoint
  50.     saver.save(sess, latest_path)
  51.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1173, in save
  52.     {self.saver_def.filename_tensor_name: checkpoint_file})
  53.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 950, in run
  54.     run_metadata_ptr)
  55.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1173, in _run
  56.     feed_dict_tensor, options, run_metadata)
  57.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1350, in _do_run
  58.     run_metadata)
  59.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1370, in _do_call
  60.     raise type(e)(node_def, op, message)
  61. tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation w/Initializer/random_normal/mul: Could not satisfy explicit device specification '' because the node node w/Initializer/random_normal/mul (defined at training.py:90) placed on device Device assignments active during op 'w/Initializer/random_normal/mul' creation:
  62.   with tf.device(None): </usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py:602>  was colocated with a group of nodes that required incompatible device '/job:localhost/replica:0/task:0/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:0, /job:localhost/replica:0/task:0/device:GPU:0].
  63. Colocation Debug Info:
  64. Colocation group had the following types and supported devices:
  65. Root Member(assigned_device_name_index_=1 requested_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' assigned_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' resource_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
  66. UnsortedSegmentSum: GPU CPU XLA_CPU XLA_GPU
  67. ResourceGather: GPU CPU XLA_CPU XLA_GPU
  68. Shape: GPU CPU XLA_CPU XLA_GPU
  69. Unique: GPU CPU
  70. ReadVariableOp: GPU CPU XLA_CPU XLA_GPU
  71. ResourceSparseApplyAdagrad: CPU
  72. StridedSlice: GPU CPU XLA_CPU XLA_GPU
  73. AssignVariableOp: GPU CPU XLA_CPU XLA_GPU
  74. Identity: GPU CPU XLA_CPU XLA_GPU
  75. RandomStandardNormal: GPU CPU XLA_CPU XLA_GPU
  76. Mul: GPU CPU XLA_CPU XLA_GPU
  77. Add: GPU CPU XLA_CPU XLA_GPU
  78. VarHandleOp: GPU CPU XLA_CPU XLA_GPU
  79. Const: GPU CPU XLA_CPU XLA_GPU
  80. VarIsInitializedOp: GPU CPU XLA_CPU XLA_GPU
  81.  
  82. Colocation members, user-requested devices, and framework assigned devices, if any:
  83.   w/Initializer/random_normal/shape (Const)
  84.   w/Initializer/random_normal/mean (Const)
  85.   w/Initializer/random_normal/stddev (Const)
  86.   w/Initializer/random_normal/RandomStandardNormal (RandomStandardNormal)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  87.   w/Initializer/random_normal/mul (Mul)
  88.   w/Initializer/random_normal (Add)
  89.   w (VarHandleOp)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  90.   w/IsInitialized/VarIsInitializedOp (VarIsInitializedOp)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  91.   w/Assign (AssignVariableOp)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  92.   w/Read/ReadVariableOp (ReadVariableOp)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  93.   tied_embedding_softmax/embedding_lookup (ResourceGather)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  94.   tied_embedding_softmax/embedding_lookup/Identity (Identity)
  95.   tied_embedding_softmax_1/transpose/ReadVariableOp (ReadVariableOp)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  96.   VarIsInitializedOp_769 (VarIsInitializedOp)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  97.   AssignVariableOp (AssignVariableOp)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  98.   ReadVariableOp (ReadVariableOp)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  99.   w/Adagrad/Initializer/Const (Const)
  100.   w/Adagrad (VarHandleOp)
  101.   w/Adagrad/IsInitialized/VarIsInitializedOp (VarIsInitializedOp)
  102.   w/Adagrad/Assign (AssignVariableOp)
  103.   w/Adagrad/Read/ReadVariableOp (ReadVariableOp)
  104.   training/Adagrad/update_w/Unique (Unique)
  105.   training/Adagrad/update_w/Shape (Shape)
  106.   training/Adagrad/update_w/strided_slice/stack (Const)
  107.   training/Adagrad/update_w/strided_slice/stack_1 (Const)
  108.   training/Adagrad/update_w/strided_slice/stack_2 (Const)
  109.   training/Adagrad/update_w/strided_slice (StridedSlice)
  110.   training/Adagrad/update_w/UnsortedSegmentSum (UnsortedSegmentSum)
  111.   training/Adagrad/update_w/ResourceSparseApplyAdagrad (ResourceSparseApplyAdagrad)
  112.   save/AssignVariableOp_1542 (AssignVariableOp)
  113.   save/AssignVariableOp_1543 (AssignVariableOp)
  114.  
  115.          [[node w/Initializer/random_normal/mul (defined at training.py:90) ]]Additional information about colocations:No node-device colocations were active during op 'w/Initializer/random_normal/mul' creation.
  116. Device assignments active during op 'w/Initializer/random_normal/mul' creation:
  117.   with tf.device(None): </usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py:602>
  118.  
  119. Original stack trace for u'w/Initializer/random_normal/mul':
  120.   File "training.py", line 162, in <module>
  121.     estimator_model = tf.keras.estimator.model_to_estimator(keras_model=model, config=run_config)
  122.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/estimator/__init__.py", line 73, in model_to_estimator
  123.     config=config)
  124.   File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/keras.py", line 450, in model_to_estimator
  125.     config)
  126.   File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/keras.py", line 318, in _save_first_checkpoint
  127.     custom_objects)
  128.   File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/keras.py", line 201, in _clone_and_build_model
  129.     optimizer_iterations=global_step)
  130.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/models.py", line 538, in clone_and_build_model
  131.     clone = clone_model(model, input_tensors=input_tensors)
  132.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/models.py", line 326, in clone_model
  133.     model, input_tensors=input_tensors, layer_fn=clone_function)
  134.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/models.py", line 154, in _clone_functional_model
  135.     new_layer = layer_fn(layer)
  136.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/models.py", line 54, in _clone_layer
  137.     return layer.__class__.from_config(layer.get_config())
  138.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 446, in from_config
  139.     return cls(**config)
  140.   File "training.py", line 90, in __init__
  141.     trainable=True)
  142.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 384, in add_weight
  143.     aggregation=aggregation)
  144.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/tracking/base.py", line 663, in _add_variable_with_custom_getter
  145.     **kwargs_for_getter)
  146.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 155, in make_variable
  147.     shape=variable_shape if variable_shape.rank else None)
  148.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 259, in __call__
  149.     return cls._variable_v1_call(*args, **kwargs)
  150.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 220, in _variable_v1_call
  151.     shape=shape)
  152.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 198, in <lambda>
  153.     previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  154.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 2495, in default_variable_creator
  155.     shape=shape)
  156.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 263, in __call__
  157.     return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  158.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 460, in __init__
  159.     shape=shape)
  160.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 604, in _init_from_args
  161.     initial_value() if init_from_fn else initial_value,
  162.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 135, in <lambda>
  163.     init_val = lambda: initializer(shape, dtype=dtype)
  164.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/init_ops.py", line 323, in __call__
  165.     shape, self.mean, self.stddev, dtype, seed=self.seed)
  166.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/random_ops.py", line 80, in random_normal
  167.     mul = rnd * stddev_tensor
  168.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 884, in binary_op_wrapper
  169.     return func(x, y, name=name)
  170.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 1180, in _mul_dispatch
  171.     return gen_math_ops.mul(x, y, name=name)
  172.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 6490, in mul
  173.     "Mul", x=x, y=y, name=name)
  174.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
  175.     op_def=op_def)
  176.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
  177.     return func(*args, **kwargs)
  178.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
  179.     op_def=op_def)
  180.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
  181.     self._traceback = tf_stack.extract_stack()
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!