Advertisement
Guest User

Error

a guest
Dec 10th, 2019
208
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 6.67 KB | None | 0 0
  1. Traceback (most recent call last):
  2. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
  3. return fn(*args)
  4. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
  5. target_list, run_metadata)
  6. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
  7. run_metadata)
  8. tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  9. (0) Resource exhausted: OOM when allocating tensor with shape[1,48,2,25,582,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
  10. [[{{node sample_sequence/while/concat}}]]
  11. Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
  12.  
  13. [[sample_sequence/while/Exit_3/_1387]]
  14. Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
  15.  
  16. (1) Resource exhausted: OOM when allocating tensor with shape[1,48,2,25,582,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
  17. [[{{node sample_sequence/while/concat}}]]
  18. Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
  19.  
  20. 0 successful operations.
  21. 0 derived errors ignored.
  22.  
  23. During handling of the above exception, another exception occurred:
  24.  
  25. Traceback (most recent call last):
  26. File "play.py", line 271, in <module>
  27. play_aidungeon_2()
  28. File "play.py", line 234, in play_aidungeon_2
  29. result = "\n" + story_manager.act(action)
  30. File "/content/AIDungeon/AIDungeon/AIDungeon/story/story_manager.py", line 207, in act
  31. result = self.generate_result(action_choice)
  32. File "/content/AIDungeon/AIDungeon/AIDungeon/story/story_manager.py", line 212, in generate_result
  33. block = self.generator.generate(self.story_context() + action)
  34. File "/content/AIDungeon/AIDungeon/AIDungeon/generator/gpt2/gpt2_generator.py", line 119, in generate
  35. text = self.generate_raw(prompt)
  36. File "/content/AIDungeon/AIDungeon/AIDungeon/generator/gpt2/gpt2_generator.py", line 102, in generate_raw
  37. self.context: [context_tokens for _ in range(self.batch_size)]
  38. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 956, in run
  39. run_metadata_ptr)
  40. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1180, in _run
  41. feed_dict_tensor, options, run_metadata)
  42. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
  43. run_metadata)
  44. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
  45. raise type(e)(node_def, op, message)
  46. tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  47. (0) Resource exhausted: OOM when allocating tensor with shape[1,48,2,25,582,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
  48. [[node sample_sequence/while/concat (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
  49. Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
  50.  
  51. [[sample_sequence/while/Exit_3/_1387]]
  52. Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
  53.  
  54. (1) Resource exhausted: OOM when allocating tensor with shape[1,48,2,25,582,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
  55. [[node sample_sequence/while/concat (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
  56. Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
  57.  
  58. 0 successful operations.
  59. 0 derived errors ignored.
  60.  
  61. Original stack trace for 'sample_sequence/while/concat':
  62. File "play.py", line 271, in <module>
  63. play_aidungeon_2()
  64. File "play.py", line 106, in play_aidungeon_2
  65. generator = GPT2Generator()
  66. File "/content/AIDungeon/AIDungeon/AIDungeon/generator/gpt2/gpt2_generator.py", line 51, in __init__
  67. top_p=top_p,
  68. File "/content/AIDungeon/AIDungeon/AIDungeon/generator/gpt2/src/sample.py", line 120, in sample_sequence
  69. back_prop=False,
  70. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/control_flow_ops.py", line 2753, in while_loop
  71. return_same_structure)
  72. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/control_flow_ops.py", line 2245, in BuildLoop
  73. pred, body, original_loop_vars, loop_vars, shape_invariants)
  74. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/control_flow_ops.py", line 2170, in _BuildLoop
  75. body_result = body(*packed_vars_for_body)
  76. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/control_flow_ops.py", line 2705, in <lambda>
  77. body = lambda i, lv: (i + 1, orig_body(*lv))
  78. File "/content/AIDungeon/AIDungeon/AIDungeon/generator/gpt2/src/sample.py", line 98, in body
  79. else tf.concat([past, next_outputs["presents"]], axis=-2),
  80. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper
  81. return target(*args, **kwargs)
  82. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/array_ops.py", line 1420, in concat
  83. return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
  84. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_array_ops.py", line 1257, in concat_v2
  85. "ConcatV2", values=values, axis=axis, name=name)
  86. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
  87. op_def=op_def)
  88. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
  89. return func(*args, **kwargs)
  90. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
  91. attrs, op_def, compute_device)
  92. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
  93. op_def=op_def)
  94. File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
  95. self._traceback = tf_stack.extract_stack()
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement