Advertisement
nickwalton

error 2

Oct 22nd, 2019
114
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Bash 14.76 KB | None | 0 0
  1. lder_760: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  2. Placeholder_761: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  3. 2019-10-22 20:19:23.773806: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_761: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  4. Placeholder_762: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  5. 2019-10-22 20:19:23.773827: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_762: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  6. Placeholder_763: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  7. 2019-10-22 20:19:23.773846: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_763: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  8. Placeholder_764: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  9. 2019-10-22 20:19:23.773865: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_764: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  10. Placeholder_765: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  11. 2019-10-22 20:19:23.773885: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_765: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  12. Placeholder_766: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  13. 2019-10-22 20:19:23.773906: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_766: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  14. Placeholder_767: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  15. 2019-10-22 20:19:23.773926: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_767: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  16. Placeholder_768: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  17. 2019-10-22 20:19:23.773947: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_768: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  18. Placeholder_769: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  19. 2019-10-22 20:19:23.773967: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_769: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  20. Placeholder_770: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  21. 2019-10-22 20:19:23.773988: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_770: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  22. Placeholder_771: (Placeholder): /job:localhost/replica:0/task:0/device:GPU:0
  23. 2019-10-22 20:19:23.774009: I tensorflow/core/common_runtime/placer.cc:54] Placeholder_771: (Placeholder)/job:localhost/replica:0/task:0/device:GPU:0
  24. W1022 20:19:57.899576 140631994844992 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/clip_ops.py:286: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
  25. Instructions for updating:
  26. Use tf.where in 2.0, which has the same broadcast rule as np.where
  27. W1022 20:19:58.752142 140631994844992 deprecation.py:506] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/adagrad.py:76: calling __init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
  28. Instructions for updating:
  29. Call initializer instance with the dtype argument instead of passing it to the constructor
  30. global_step: (VariableV2): /job:localhost/replica:0/task:0/device:GPU:0
  31. 2019-10-22 20:20:08.083224: I tensorflow/core/common_runtime/placer.cc:54] global_step: (VariableV2)/job:localhost/replica:0/task:0/device:GPU:0
  32. global_step/Assign: (Assign): /job:localhost/replica:0/task:0/device:GPU:0
  33. 2019-10-22 20:20:08.083279: I tensorflow/core/common_runtime/placer.cc:54] global_step/Assign: (Assign)/job:localhost/replica:0/task:0/device:GPU:0
  34. global_step/read: (Identity): /job:localhost/replica:0/task:0/device:GPU:0
  35. 2019-10-22 20:20:08.083294: I tensorflow/core/common_runtime/placer.cc:54] global_step/read: (Identity)/job:localhost/replica:0/task:0/device:GPU:0
  36. w/Initializer/random_normal/RandomStandardNormal: (RandomStandardNormal): /job:localhost/replica:0/task:0/device:GPU:0
  37. 2019-10-22 20:20:08.083308: I tensorflow/core/common_runtime/placer.cc:54] w/Initializer/random_normal/RandomStandardNormal: (RandomStandardNormal)/job:localhost/replica:0/task:0/device:GPU:0
  38. Traceback (most recent call last):
  39.   File "training.py", line 148, in <module>
  40.     estimator_model = tf.keras.estimator.model_to_estimator(keras_model=model, config=run_config)
  41.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/estimator/__init__.py", line 73, in model_to_estimator
  42.     config=config)
  43.   File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/keras.py", line 450, in model_to_estimator
  44.     config)
  45.   File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/keras.py", line 331, in _save_first_checkpoint
  46.     saver.save(sess, latest_path)
  47.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1173, in save
  48.     {self.saver_def.filename_tensor_name: checkpoint_file})
  49.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 950, in run
  50.     run_metadata_ptr)
  51.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1173, in _run
  52.     feed_dict_tensor, options, run_metadata)
  53.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1350, in _do_run
  54.     run_metadata)
  55.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1370, in _do_call
  56.     raise type(e)(node_def, op, message)
  57. tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation w/Initializer/random_normal/mul: Could not satisfy explicit device specification '' because the node node w/Initializer/random_normal/mul (defined at training.py:87) placed on device Device assignments active during op 'w/Initializer/random_normal/mul' creation:
  58.   with tf.device(None): </usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py:602>  was colocated with a group of nodes that required incompatible device '/job:localhost/replica:0/task:0/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:1, /job:localhost/replica:0/task:0/device:GPU:0, /job:localhost/replica:0/task:0/device:GPU:1].
  59. Colocation Debug Info:
  60. Colocation group had the following types and supported devices:
  61. Root Member(assigned_device_name_index_=1 requested_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' assigned_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' resource_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
  62. UnsortedSegmentSum: GPU CPU XLA_CPU XLA_GPU
  63. ResourceGather: GPU CPU XLA_CPU XLA_GPU
  64. Shape: GPU CPU XLA_CPU XLA_GPU
  65. Unique: GPU CPU
  66. ReadVariableOp: GPU CPU XLA_CPU XLA_GPU
  67. ResourceSparseApplyAdagrad: CPU
  68. StridedSlice: GPU CPU XLA_CPU XLA_GPU
  69. AssignVariableOp: GPU CPU XLA_CPU XLA_GPU
  70. Identity: GPU CPU XLA_CPU XLA_GPU
  71. RandomStandardNormal: GPU CPU XLA_CPU XLA_GPU
  72. Mul: GPU CPU XLA_CPU XLA_GPU
  73. Add: GPU CPU XLA_CPU XLA_GPU
  74. VarHandleOp: GPU CPU XLA_CPU XLA_GPU
  75. Const: GPU CPU XLA_CPU XLA_GPU
  76. VarIsInitializedOp: GPU CPU XLA_CPU XLA_GPU
  77.  
  78. Colocation members, user-requested devices, and framework assigned devices, if any:
  79.   w/Initializer/random_normal/shape (Const)
  80.   w/Initializer/random_normal/mean (Const)
  81.   w/Initializer/random_normal/stddev (Const)
  82.   w/Initializer/random_normal/RandomStandardNormal (RandomStandardNormal)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  83.   w/Initializer/random_normal/mul (Mul)
  84.   w/Initializer/random_normal (Add)
  85.   w (VarHandleOp)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  86.   w/IsInitialized/VarIsInitializedOp (VarIsInitializedOp)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  87.   w/Assign (AssignVariableOp)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  88.   w/Read/ReadVariableOp (ReadVariableOp)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  89.   tied_embedding_softmax/embedding_lookup (ResourceGather)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  90.   tied_embedding_softmax/embedding_lookup/Identity (Identity)
  91.   tied_embedding_softmax_1/transpose/ReadVariableOp (ReadVariableOp)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  92.   VarIsInitializedOp_189 (VarIsInitializedOp)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  93.   AssignVariableOp (AssignVariableOp)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  94.   ReadVariableOp (ReadVariableOp)  framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
  95.   w/Adagrad/Initializer/Const (Const)
  96.   w/Adagrad (VarHandleOp)
  97.   w/Adagrad/IsInitialized/VarIsInitializedOp (VarIsInitializedOp)
  98.   w/Adagrad/Assign (AssignVariableOp)
  99.   w/Adagrad/Read/ReadVariableOp (ReadVariableOp)
  100.   training/Adagrad/update_w/Unique (Unique)
  101.   training/Adagrad/update_w/Shape (Shape)
  102.   training/Adagrad/update_w/strided_slice/stack (Const)
  103.   training/Adagrad/update_w/strided_slice/stack_1 (Const)
  104.   training/Adagrad/update_w/strided_slice/stack_2 (Const)
  105.   training/Adagrad/update_w/strided_slice (StridedSlice)
  106.   training/Adagrad/update_w/UnsortedSegmentSum (UnsortedSegmentSum)
  107.   training/Adagrad/update_w/ResourceSparseApplyAdagrad (ResourceSparseApplyAdagrad)
  108.   save/AssignVariableOp_1542 (AssignVariableOp)
  109.   save/AssignVariableOp_1543 (AssignVariableOp)
  110.  
  111.          [[node w/Initializer/random_normal/mul (defined at training.py:87) ]]Additional information about colocations:No node-device colocations were active during op 'w/Initializer/random_normal/mul' creation.
  112. Device assignments active during op 'w/Initializer/random_normal/mul' creation:
  113.   with tf.device(None): </usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py:602>
  114.  
  115. Original stack trace for u'w/Initializer/random_normal/mul':
  116.   File "training.py", line 148, in <module>
  117.     estimator_model = tf.keras.estimator.model_to_estimator(keras_model=model, config=run_config)
  118.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/estimator/__init__.py", line 73, in model_to_estimator
  119.     config=config)
  120.   File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/keras.py", line 450, in model_to_estimator
  121.     config)
  122.   File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/keras.py", line 318, in _save_first_checkpoint
  123.     custom_objects)
  124.   File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/keras.py", line 201, in _clone_and_build_model
  125.     optimizer_iterations=global_step)
  126.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/models.py", line 538, in clone_and_build_model
  127.     clone = clone_model(model, input_tensors=input_tensors)
  128.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/models.py", line 326, in clone_model
  129.     model, input_tensors=input_tensors, layer_fn=clone_function)
  130.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/models.py", line 154, in _clone_functional_model
  131.     new_layer = layer_fn(layer)
  132.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/models.py", line 54, in _clone_layer
  133.     return layer.__class__.from_config(layer.get_config())
  134.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 446, in from_config
  135.     return cls(**config)
  136.   File "training.py", line 87, in __init__
  137.     trainable=True)
  138.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 384, in add_weight
  139.     aggregation=aggregation)
  140.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/tracking/base.py", line 663, in _add_variable_with_custom_getter
  141.     **kwargs_for_getter)
  142.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 155, in make_variable
  143.     shape=variable_shape if variable_shape.rank else None)
  144.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 259, in __call__
  145.     return cls._variable_v1_call(*args, **kwargs)
  146.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 220, in _variable_v1_call
  147.     shape=shape)
  148.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 198, in <lambda>
  149.     previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  150.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 2495, in default_variable_creator
  151.     shape=shape)
  152.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 263, in __call__
  153.     return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  154.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 460, in __init__
  155.     shape=shape)
  156.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 604, in _init_from_args
  157.     initial_value() if init_from_fn else initial_value,
  158.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 135, in <lambda>
  159.     init_val = lambda: initializer(shape, dtype=dtype)
  160.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/init_ops.py", line 323, in __call__
  161.     shape, self.mean, self.stddev, dtype, seed=self.seed)
  162.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/random_ops.py", line 80, in random_normal
  163.     mul = rnd * stddev_tensor
  164.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 884, in binary_op_wrapper
  165.     return func(x, y, name=name)
  166.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 1180, in _mul_dispatch
  167.     return gen_math_ops.mul(x, y, name=name)
  168.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 6490, in mul
  169.     "Mul", x=x, y=y, name=name)
  170.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
  171.     op_def=op_def)
  172.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
  173.     return func(*args, **kwargs)
  174.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
  175.     op_def=op_def)
  176.   File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
  177.     self._traceback = tf_stack.extract_stack()
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement