API tools faq
paste
Guest User

after triton reinstall

a guest
Dec 2nd, 2023
331
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 85.41 KB | None | 0 0
raw download clone embed print report
  1. Loading config: small
  2. project path: /home/robert/languini-kitchen/languini/projects/moe
  3. Arguments:
  4. data_root=data/books
  5. relative_log_path=logs
  6. dataset=books_16384
  7. vocab_size=16384
  8. debug=False
  9. seed=0
  10. gradient_accumulation_steps=1
  11. train_batch_size=1
  12. overwriting train_batch_size to 1
  13. eval_batch_size=16
  14. test_batch_size=16
  15. seq_len=512
  16. max_eval_steps=500
  17. max_train_steps=500000
  18. decay_steps=500000
  19. max_lr=0.0006
  20. min_lr=6e-06
  21. grad_clip_norm=0.0
  22. tokens_per_second=0
  23. eval_every=1000
  24. log_terminal_every=100
  25. log_metrics_every=100
  26. log_grads_every=1000
  27. log_activations_every=-1
  28. log_ckpt_every=5000
  29. logger_type=all
  30. wandb_project_name=gpt
  31. use_flash=False
  32. n_layers=12
  33. h_dim=768
  34. mlp_dim=3072
  35. head_dim=64
  36. n_heads=12
  37. project_path=/home/robert/languini-kitchen/languini/projects/moe
  38. compile=default
  39. results_pickle_file=None
  40. would be experiment name: GPTMoE_books16384_bsz1_sl512_coslr0.0006to6e-06_h768_ff3072_nH12_dH64_nl12_clip0.0_decay500k_workers1_defaultCompile_fp16_seed0
  41. [2023-12-01 15:15:10,458] [2/0] torch._dynamo.variables.higher_order_ops: [WARNING] speculate_subgraph: while introspecting the user-defined autograd.Function, we were unable to trace function `trampoline_autograd_bwd` into a single graph. This means that Dynamo was unable to prove safety for this API and will fall back to eager-mode PyTorch, which could lead to a slowdown.
  42. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] Illegal getattr invocation stride in strict mode
  43. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] Traceback (most recent call last):
  44. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/higher_order_ops.py", line 240, in speculate_subgraph
  45. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] output = f.call_function(tx, args, sub_kwargs)
  46. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  47. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  48. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  49. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  50. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  51. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  52. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  53. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  54. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  55. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  56. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  57. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  58. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  59. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  60. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  61. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  62. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1252, in CALL_FUNCTION_EX
  63. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, argsvars.items, kwargsvars.items)
  64. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  65. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  66. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 643, in call_function
  67. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return self.obj.call_method(tx, self.name, args, kwargs)
  68. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 488, in call_method
  69. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_call(tx, backward, args, kwargs)
  70. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  71. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  72. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  73. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  74. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  75. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  76. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  77. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  78. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  79. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  80. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1264, in CALL_FUNCTION_KW
  81. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, args, kwargs)
  82. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  83. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  84. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  85. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  86. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  87. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  88. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  89. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  90. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  91. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  92. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  93. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  94. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  95. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  96. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  97. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  98. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1303, in LOAD_ATTR
  99. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = BuiltinVariable(getattr).call_function(
  100. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 636, in call_function
  101. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = handler(tx, *args, **kwargs)
  102. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1167, in call_getattr
  103. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return obj.var_getattr(tx, name).clone(source=source)
  104. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 218, in var_getattr
  105. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] unimplemented(f"Illegal getattr invocation {name} in strict mode")
  106. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 193, in unimplemented
  107. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] raise Unsupported(msg)
  108. [2023-12-01 15:15:10,459] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] torch._dynamo.exc.Unsupported: Illegal getattr invocation stride in strict mode
  109. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [WARNING] speculate_subgraph: while introspecting the user-defined autograd.Function, we were unable to trace function `trampoline_autograd_bwd` into a single graph. This means that Dynamo was unable to prove safety for this API and will fall back to eager-mode PyTorch, which could lead to a slowdown.
  110. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] Illegal getattr invocation stride in strict mode
  111. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] Traceback (most recent call last):
  112. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/higher_order_ops.py", line 240, in speculate_subgraph
  113. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] output = f.call_function(tx, args, sub_kwargs)
  114. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  115. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  116. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  117. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  118. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  119. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  120. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  121. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  122. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  123. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  124. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  125. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  126. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  127. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  128. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  129. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  130. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1252, in CALL_FUNCTION_EX
  131. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, argsvars.items, kwargsvars.items)
  132. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  133. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  134. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 643, in call_function
  135. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return self.obj.call_method(tx, self.name, args, kwargs)
  136. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 488, in call_method
  137. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_call(tx, backward, args, kwargs)
  138. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  139. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  140. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  141. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  142. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  143. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  144. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  145. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  146. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  147. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  148. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1264, in CALL_FUNCTION_KW
  149. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, args, kwargs)
  150. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  151. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  152. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  153. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  154. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  155. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  156. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  157. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  158. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  159. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  160. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  161. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  162. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  163. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  164. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  165. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  166. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1303, in LOAD_ATTR
  167. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = BuiltinVariable(getattr).call_function(
  168. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 636, in call_function
  169. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = handler(tx, *args, **kwargs)
  170. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1167, in call_getattr
  171. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return obj.var_getattr(tx, name).clone(source=source)
  172. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 218, in var_getattr
  173. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] unimplemented(f"Illegal getattr invocation {name} in strict mode")
  174. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 193, in unimplemented
  175. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] raise Unsupported(msg)
  176. [2023-12-01 15:15:13,055] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] torch._dynamo.exc.Unsupported: Illegal getattr invocation stride in strict mode
  177. [2023-12-01 15:15:13,186] [4/0] torch._dynamo.variables.higher_order_ops: [WARNING] speculate_subgraph: while introspecting the user-defined autograd.Function, we were unable to trace function `trampoline_autograd_bwd` into a single graph. This means that Dynamo was unable to prove safety for this API and will fall back to eager-mode PyTorch, which could lead to a slowdown.
  178. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] Illegal getattr invocation stride in strict mode
  179. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] Traceback (most recent call last):
  180. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/higher_order_ops.py", line 240, in speculate_subgraph
  181. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] output = f.call_function(tx, args, sub_kwargs)
  182. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  183. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  184. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  185. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  186. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  187. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  188. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  189. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  190. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  191. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  192. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  193. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  194. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  195. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  196. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  197. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  198. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1252, in CALL_FUNCTION_EX
  199. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, argsvars.items, kwargsvars.items)
  200. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  201. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  202. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 643, in call_function
  203. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return self.obj.call_method(tx, self.name, args, kwargs)
  204. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 488, in call_method
  205. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_call(tx, backward, args, kwargs)
  206. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  207. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  208. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  209. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  210. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  211. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  212. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  213. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  214. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  215. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  216. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1264, in CALL_FUNCTION_KW
  217. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, args, kwargs)
  218. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  219. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  220. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  221. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  222. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  223. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  224. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  225. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  226. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  227. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  228. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  229. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  230. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  231. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  232. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  233. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  234. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1303, in LOAD_ATTR
  235. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = BuiltinVariable(getattr).call_function(
  236. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 636, in call_function
  237. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = handler(tx, *args, **kwargs)
  238. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1167, in call_getattr
  239. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return obj.var_getattr(tx, name).clone(source=source)
  240. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 218, in var_getattr
  241. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] unimplemented(f"Illegal getattr invocation {name} in strict mode")
  242. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 193, in unimplemented
  243. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] raise Unsupported(msg)
  244. [2023-12-01 15:15:13,187] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] torch._dynamo.exc.Unsupported: Illegal getattr invocation stride in strict mode
  245. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [WARNING] speculate_subgraph: while introspecting the user-defined autograd.Function, we were unable to trace function `trampoline_autograd_bwd` into a single graph. This means that Dynamo was unable to prove safety for this API and will fall back to eager-mode PyTorch, which could lead to a slowdown.
  246. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] Illegal getattr invocation stride in strict mode
  247. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] Traceback (most recent call last):
  248. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/higher_order_ops.py", line 240, in speculate_subgraph
  249. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] output = f.call_function(tx, args, sub_kwargs)
  250. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  251. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  252. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  253. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  254. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  255. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  256. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  257. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  258. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  259. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  260. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  261. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  262. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  263. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  264. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  265. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  266. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1252, in CALL_FUNCTION_EX
  267. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, argsvars.items, kwargsvars.items)
  268. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  269. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  270. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 643, in call_function
  271. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return self.obj.call_method(tx, self.name, args, kwargs)
  272. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 488, in call_method
  273. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_call(tx, backward, args, kwargs)
  274. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  275. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  276. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  277. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  278. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  279. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  280. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  281. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  282. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  283. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  284. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1264, in CALL_FUNCTION_KW
  285. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, args, kwargs)
  286. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  287. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  288. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  289. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  290. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  291. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  292. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  293. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  294. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  295. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  296. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  297. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  298. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  299. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  300. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  301. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  302. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1303, in LOAD_ATTR
  303. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = BuiltinVariable(getattr).call_function(
  304. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 636, in call_function
  305. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = handler(tx, *args, **kwargs)
  306. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1167, in call_getattr
  307. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return obj.var_getattr(tx, name).clone(source=source)
  308. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 218, in var_getattr
  309. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] unimplemented(f"Illegal getattr invocation {name} in strict mode")
  310. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 193, in unimplemented
  311. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] raise Unsupported(msg)
  312. [2023-12-01 15:15:14,525] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] torch._dynamo.exc.Unsupported: Illegal getattr invocation stride in strict mode
  313. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [WARNING] speculate_subgraph: while introspecting the user-defined autograd.Function, we were unable to trace function `trampoline_autograd_bwd` into a single graph. This means that Dynamo was unable to prove safety for this API and will fall back to eager-mode PyTorch, which could lead to a slowdown.
  314. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] Illegal getattr invocation stride in strict mode
  315. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] Traceback (most recent call last):
  316. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/higher_order_ops.py", line 240, in speculate_subgraph
  317. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] output = f.call_function(tx, args, sub_kwargs)
  318. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  319. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  320. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  321. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  322. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  323. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  324. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  325. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  326. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  327. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  328. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  329. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  330. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  331. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  332. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  333. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  334. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1252, in CALL_FUNCTION_EX
  335. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, argsvars.items, kwargsvars.items)
  336. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  337. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  338. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 643, in call_function
  339. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return self.obj.call_method(tx, self.name, args, kwargs)
  340. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 488, in call_method
  341. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_call(tx, backward, args, kwargs)
  342. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  343. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  344. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  345. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  346. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  347. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  348. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  349. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  350. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  351. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  352. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1264, in CALL_FUNCTION_KW
  353. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, args, kwargs)
  354. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  355. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  356. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  357. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  358. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  359. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  360. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  361. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  362. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  363. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  364. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  365. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  366. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  367. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  368. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  369. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  370. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1303, in LOAD_ATTR
  371. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = BuiltinVariable(getattr).call_function(
  372. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 636, in call_function
  373. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = handler(tx, *args, **kwargs)
  374. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1167, in call_getattr
  375. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return obj.var_getattr(tx, name).clone(source=source)
  376. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 218, in var_getattr
  377. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] unimplemented(f"Illegal getattr invocation {name} in strict mode")
  378. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 193, in unimplemented
  379. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] raise Unsupported(msg)
  380. [2023-12-01 15:15:14,613] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] torch._dynamo.exc.Unsupported: Illegal getattr invocation stride in strict mode
  381. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [WARNING] speculate_subgraph: while introspecting the user-defined autograd.Function, we were unable to trace function `trampoline_autograd_bwd` into a single graph. This means that Dynamo was unable to prove safety for this API and will fall back to eager-mode PyTorch, which could lead to a slowdown.
  382. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] Illegal getattr invocation stride in strict mode
  383. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] Traceback (most recent call last):
  384. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/higher_order_ops.py", line 240, in speculate_subgraph
  385. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] output = f.call_function(tx, args, sub_kwargs)
  386. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  387. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  388. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  389. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  390. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  391. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  392. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  393. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  394. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  395. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  396. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  397. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  398. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  399. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  400. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  401. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  402. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1252, in CALL_FUNCTION_EX
  403. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, argsvars.items, kwargsvars.items)
  404. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  405. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  406. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 643, in call_function
  407. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] return self.obj.call_method(tx, self.name, args, kwargs)
  408. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 488, in call_method
  409. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_call(tx, backward, args, kwargs)
  410. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  411. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  412. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  413. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  414. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  415. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  416. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  417. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  418. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  419. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  420. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1264, in CALL_FUNCTION_KW
  421. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, args, kwargs)
  422. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  423. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  424. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  425. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  426. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  427. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  428. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  429. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  430. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  431. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  432. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  433. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  434. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  435. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  436. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  437. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  438. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1303, in LOAD_ATTR
  439. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = BuiltinVariable(getattr).call_function(
  440. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 636, in call_function
  441. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = handler(tx, *args, **kwargs)
  442. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1167, in call_getattr
  443. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] return obj.var_getattr(tx, name).clone(source=source)
  444. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 218, in var_getattr
  445. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] unimplemented(f"Illegal getattr invocation {name} in strict mode")
  446. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 193, in unimplemented
  447. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] raise Unsupported(msg)
  448. [2023-12-01 15:15:28,103] [9/0] torch._dynamo.variables.higher_order_ops: [ERROR] torch._dynamo.exc.Unsupported: Illegal getattr invocation stride in strict mode
  449. [2023-12-01 15:15:28,249] [6/1] torch._dynamo.variables.higher_order_ops: [WARNING] speculate_subgraph: while introspecting the user-defined autograd.Function, we were unable to trace function `trampoline_autograd_bwd` into a single graph. This means that Dynamo was unable to prove safety for this API and will fall back to eager-mode PyTorch, which could lead to a slowdown.
  450. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] Illegal getattr invocation stride in strict mode
  451. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] Traceback (most recent call last):
  452. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/higher_order_ops.py", line 240, in speculate_subgraph
  453. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] output = f.call_function(tx, args, sub_kwargs)
  454. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  455. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  456. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  457. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  458. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  459. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  460. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  461. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  462. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  463. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  464. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  465. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  466. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  467. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  468. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  469. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  470. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1252, in CALL_FUNCTION_EX
  471. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, argsvars.items, kwargsvars.items)
  472. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  473. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  474. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 643, in call_function
  475. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] return self.obj.call_method(tx, self.name, args, kwargs)
  476. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 488, in call_method
  477. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_call(tx, backward, args, kwargs)
  478. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  479. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  480. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  481. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  482. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  483. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  484. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  485. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  486. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  487. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  488. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1264, in CALL_FUNCTION_KW
  489. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, args, kwargs)
  490. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  491. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  492. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  493. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  494. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  495. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  496. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  497. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  498. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  499. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  500. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  501. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  502. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  503. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  504. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  505. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  506. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1303, in LOAD_ATTR
  507. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] result = BuiltinVariable(getattr).call_function(
  508. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 636, in call_function
  509. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] result = handler(tx, *args, **kwargs)
  510. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1167, in call_getattr
  511. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] return obj.var_getattr(tx, name).clone(source=source)
  512. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 218, in var_getattr
  513. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] unimplemented(f"Illegal getattr invocation {name} in strict mode")
  514. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 193, in unimplemented
  515. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] raise Unsupported(msg)
  516. [2023-12-01 15:15:28,250] [6/1] torch._dynamo.variables.higher_order_ops: [ERROR] torch._dynamo.exc.Unsupported: Illegal getattr invocation stride in strict mode
  517. Traceback (most recent call last):
  518. File "/home/robert/languini-kitchen/languini/common_lib/throughput.py", line 132, in <module>
  519. main()
  520. File "/home/robert/languini-kitchen/languini/common_lib/throughput.py", line 82, in main
  521. result = throughput_utils.throughput_test(config=config, model=model)
  522. File "/home/robert/languini-kitchen/languini/common_lib/throughput_utils.py", line 79, in throughput_test
  523. step(inputs, targets, state)
  524. File "/home/robert/languini-kitchen/languini/common_lib/throughput_utils.py", line 65, in step
  525. logits, _ = model(inputs, state)
  526. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
  527. return self._call_impl(*args, **kwargs)
  528. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
  529. return forward_call(*args, **kwargs)
  530. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 489, in _fn
  531. return fn(*args, **kwargs)
  532. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
  533. return self._call_impl(*args, **kwargs)
  534. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
  535. return forward_call(*args, **kwargs)
  536. File "/home/robert/languini-kitchen/languini/projects/moe/model.py", line 91, in forward
  537. x = layer(x, log=log)
  538. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
  539. return self._call_impl(*args, **kwargs)
  540. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
  541. return forward_call(*args, **kwargs)
  542. File "/home/robert/languini-kitchen/languini/projects/moe/lib.py", line 190, in forward
  543. mlp_x = self.mlp(self.ln2(x, log=log), log=log)
  544. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
  545. return self._call_impl(*args, **kwargs)
  546. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
  547. return forward_call(*args, **kwargs)
  548. File "/home/robert/languini-kitchen/languini/projects/moe/lib.py", line 159, in forward
  549. x, reg_loss = self.moe(x)
  550. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
  551. return self._call_impl(*args, **kwargs)
  552. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
  553. return forward_call(*args, **kwargs)
  554. File "/home/robert/languini-kitchen/languini/projects/moe/moe_layer/moe_layer_simple.py", line 166, in forward
  555. scores = self.compute_scores(input, sel_indices)
  556. File "/home/robert/languini-kitchen/languini/projects/moe/moe_layer/moe_layer_simple.py", line 174, in resume_in_forward
  557. out = cvmm(scores, sel_indices, self.values)
  558. File "/home/robert/languini-kitchen/languini/projects/moe/moe_layer/cvmm.py", line 473, in cvmm
  559. return CVMM.apply(x, sel.sel_index, sel.sel, keys, sel.out_index, sel.reduction_weight)
  560. File "/home/robert/.local/lib/python3.10/site-packages/torch/autograd/function.py", line 553, in apply
  561. return super().apply(*args, **kwargs) # type: ignore[misc]
  562. File "/home/robert/languini-kitchen/languini/projects/moe/moe_layer/cvmm.py", line 401, in forward
  563. @staticmethod
  564. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 489, in _fn
  565. return fn(*args, **kwargs)
  566. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/external_utils.py", line 17, in inner
  567. return fn(*args, **kwargs)
  568. File "/home/robert/.local/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 899, in forward
  569. return compiled_fn(full_args)
  570. File "/home/robert/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 81, in g
  571. return f(*args)
  572. File "/home/robert/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 94, in runtime_wrapper
  573. all_outs = call_func_at_runtime_with_args(
  574. File "/home/robert/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 105, in call_func_at_runtime_with_args
  575. out = normalize_as_list(f(args))
  576. File "/home/robert/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 118, in rng_functionalization_wrapper
  577. return compiled_fw(args)
  578. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 861, in __call__
  579. return self.get_current_callable()(inputs)
  580. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 611, in run
  581. return model(new_inputs)
  582. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 889, in _run_from_cache
  583. return compiled_graph.compiled_artifact(inputs)
  584. File "/tmp/torchinductor_robert/us/cus2tttgmlum26bo6gt62owskybg756jj26yzdp4pmrybxzrhh4q.py", line 316, in call
  585. triton_poi_fused_2.run(buf1, arg3_1, triton_poi_fused_2_xnumel, grid=grid(triton_poi_fused_2_xnumel), stream=stream0)
  586. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 533, in run
  587. self.autotune_to_one_config(*args, grid=grid, **kwargs)
  588. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 437, in autotune_to_one_config
  589. timings = self.benchmark_all_configs(*args, **kwargs)
  590. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
  591. r = func(*args, **kwargs)
  592. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 413, in benchmark_all_configs
  593. timings = {
  594. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 414, in <dictcomp>
  595. launcher: self.bench(launcher, *args, **kwargs)
  596. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 385, in bench
  597. return do_bench(kernel_call, rep=40, fast_flush=True)
  598. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/utils.py", line 167, in do_bench
  599. return triton_do_bench(*args, **kwargs)[0]
  600. File "/home/robert/.local/lib/python3.10/site-packages/triton/testing.py", line 105, in do_bench
  601. torch.cuda.synchronize()
  602. File "/home/robert/.local/lib/python3.10/site-packages/torch/cuda/__init__.py", line 801, in synchronize
  603. return torch._C._cuda_synchronize()
  604. RuntimeError: CUDA error: an illegal memory access was encountered
  605. CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
  606. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
  607. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!