API tools faq
paste
Advertisement
Guest User

Glow invertibility crash

a guest
Jul 28th, 2018
354
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 7.64 KB | None | 0 0
raw download clone embed print report
  1. ...
  2. 11680 1.4699468612670898 [2.8573072 2.8573072 0. 1. ]
  3. 11696 1.5704457759857178 [1.9587295 1.9587295 0. 1. ]
  4. 11712 1.4362401962280273 [5.5618033 5.5618033 0. 1. ]
  5. Traceback (most recent call last):
  6. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
  7. return fn(*args)
  8. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
  9. options, feed_dict, fetch_list, target_list, run_metadata)
  10. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
  11. run_metadata)
  12. tensorflow.python.framework.errors_impl.InvalidArgumentError: Input is not invertible.
  13. [[Node: gradients_57/model/1/12/invconv/MatrixDeterminant_1_grad/MatrixInverse = MatrixInverse[T=DT_DOUBLE, adjoint=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/1/12/invconv/Cast_2)]]
  14. [[Node: gradients_73/model/0/10/add_1_grad/Shape_1/_30467 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_79355_gradients_73/model/0/10/add_1_grad/Shape_1", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
  15.  
  16. During handling of the above exception, another exception occurred:
  17.  
  18. Traceback (most recent call last):
  19. File "train.py", line 417, in <module>
  20. main(hps)
  21. File "train.py", line 163, in main
  22. train(sess, model, hps, logdir, visualise)
  23. File "train.py", line 226, in train
  24. train_results += [model.train(lr)]
  25. File "/home/gwern/src/glow/model.py", line 39, in <lambda>
  26. m.train = lambda _lr: sess.run([train_op, stats_train], {lr: _lr})[1]
  27. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
  28. run_metadata_ptr)
  29. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
  30. feed_dict_tensor, options, run_metadata)
  31. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
  32. run_metadata)
  33. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
  34. raise type(e)(node_def, op, message)
  35. tensorflow.python.framework.errors_impl.InvalidArgumentError: Input is not invertible.
  36. [[Node: gradients_57/model/1/12/invconv/MatrixDeterminant_1_grad/MatrixInverse = MatrixInverse[T=DT_DOUBLE, adjoint=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/1/12/invconv/Cast_2)]]
  37. [[Node: gradients_73/model/0/10/add_1_grad/Shape_1/_30467 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_79355_gradients_73/model/0/10/add_1_grad/Shape_1", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
  38.  
  39. Caused by op 'gradients_57/model/1/12/invconv/MatrixDeterminant_1_grad/MatrixInverse', defined at:
  40. File "train.py", line 417, in <module>
  41. main(hps)
  42. File "train.py", line 156, in main
  43. model = model.model(sess, hps, train_iterator, test_iterator, data_init)
  44. File "/home/gwern/src/glow/model.py", line 223, in model
  45. test_iterator, data_init, lr, f_loss)
  46. File "/home/gwern/src/glow/model.py", line 29, in abstract_model_xy
  47. gs = gradients(loss_train, all_params)
  48. File "/home/gwern/src/glow/memory_saving_gradients.py", line 279, in gradients
  49. grad_ys=substitute_backprops, **kwargs)
  50. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 494, in gradients
  51. gate_gradients, aggregation_method, stop_gradients)
  52. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 636, in _GradientsHelper
  53. lambda: grad_fn(op, *out_grads))
  54. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 385, in _MaybeCompile
  55. return grad_fn() # Exit early
  56. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 636, in <lambda>
  57. lambda: grad_fn(op, *out_grads))
  58. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/python/ops/linalg_grad.py", line 51, in _MatrixDeterminantGrad
  59. a_adj_inv = linalg_ops.matrix_inverse(a, adjoint=True)
  60. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/python/ops/gen_linalg_ops.py", line 1049, in matrix_inverse
  61. "MatrixInverse", input=input, adjoint=adjoint, name=name)
  62. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
  63. op_def=op_def)
  64. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
  65. op_def=op_def)
  66. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1718, in __init__
  67. self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
  68.  
  69. ...which was originally created as op 'model/1/12/invconv/MatrixDeterminant_1', defined at:
  70. File "train.py", line 417, in <module>
  71. main(hps)
  72. [elided 2 identical lines from previous traceback]
  73. File "/home/gwern/src/glow/model.py", line 29, in abstract_model_xy
  74. gs = gradients(loss_train, all_params)
  75. File "/home/gwern/src/glow/memory_saving_gradients.py", line 266, in gradients
  76. ge.sgv(ops_to_copy), {})
  77. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/contrib/graph_editor/transform.py", line 667, in copy_with_input_replacements
  78. sgv, dst_graph, dst_scope, src_scope, reuse_dst_scope=reuse_dst_scope)
  79. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/contrib/graph_editor/transform.py", line 447, in __call__
  80. self._copy_ops(info)
  81. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/contrib/graph_editor/transform.py", line 461, in _copy_ops
  82. op_, op_outputs_ = self.transform_op_handler(info, op, new_inputs)
  83. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/contrib/graph_editor/transform.py", line 168, in copy_op_handler
  84. [], input_types_, None, op_def_)
  85. File "/home/gwern/bin/miniconda2/envs/fastai/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1718, in __init__
  86. self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
  87.  
  88. InvalidArgumentError (see above for traceback): Input is not invertible.
  89. [[Node: gradients_57/model/1/12/invconv/MatrixDeterminant_1_grad/MatrixInverse = MatrixInverse[T=DT_DOUBLE, adjoint=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](model/1/12/invconv/Cast_2)]]
  90. [[Node: gradients_73/model/0/10/add_1_grad/Shape_1/_30467 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_79355_gradients_73/model/0/10/add_1_grad/Shape_1", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!