API tools faq
paste
Guest User

nighly failure

a guest
Dec 2nd, 2023
272
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 67.55 KB | None | 0 0
raw download clone embed print report
  1. Loading config: small
  2. project path: /home/robert/languini-kitchen/languini/projects/moe
  3. Arguments:
  4. data_root=data/books
  5. relative_log_path=logs
  6. dataset=books_16384
  7. vocab_size=16384
  8. debug=False
  9. seed=0
  10. gradient_accumulation_steps=1
  11. train_batch_size=1
  12. overwriting train_batch_size to 1
  13. eval_batch_size=16
  14. test_batch_size=16
  15. seq_len=512
  16. max_eval_steps=500
  17. max_train_steps=500000
  18. decay_steps=500000
  19. max_lr=0.0006
  20. min_lr=6e-06
  21. grad_clip_norm=0.0
  22. tokens_per_second=0
  23. eval_every=1000
  24. log_terminal_every=100
  25. log_metrics_every=100
  26. log_grads_every=1000
  27. log_activations_every=-1
  28. log_ckpt_every=5000
  29. logger_type=all
  30. wandb_project_name=gpt
  31. use_flash=False
  32. n_layers=12
  33. h_dim=768
  34. mlp_dim=3072
  35. head_dim=64
  36. n_heads=12
  37. project_path=/home/robert/languini-kitchen/languini/projects/moe
  38. compile=default
  39. results_pickle_file=None
  40. would be experiment name: GPTMoE_books16384_bsz1_sl512_coslr0.0006to6e-06_h768_ff3072_nH12_dH64_nl12_clip0.0_decay500k_workers1_defaultCompile_fp16_seed0
  41. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [WARNING] speculate_subgraph: while introspecting the user-defined autograd.Function, we were unable to trace function `trampoline_autograd_bwd` into a single graph. This means that Dynamo was unable to prove safety for this API and will fall back to eager-mode PyTorch, which could lead to a slowdown.
  42. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] Illegal getattr invocation stride in strict mode
  43. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] Traceback (most recent call last):
  44. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/higher_order_ops.py", line 240, in speculate_subgraph
  45. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] output = f.call_function(tx, args, sub_kwargs)
  46. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  47. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  48. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  49. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  50. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  51. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  52. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  53. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  54. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  55. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  56. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  57. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  58. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  59. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  60. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  61. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  62. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1252, in CALL_FUNCTION_EX
  63. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, argsvars.items, kwargsvars.items)
  64. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  65. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  66. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 643, in call_function
  67. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return self.obj.call_method(tx, self.name, args, kwargs)
  68. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 488, in call_method
  69. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_call(tx, backward, args, kwargs)
  70. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  71. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  72. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  73. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  74. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  75. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  76. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  77. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  78. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  79. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  80. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1264, in CALL_FUNCTION_KW
  81. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, args, kwargs)
  82. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  83. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  84. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  85. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  86. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  87. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  88. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  89. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  90. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  91. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  92. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  93. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  94. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  95. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  96. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  97. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  98. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1303, in LOAD_ATTR
  99. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = BuiltinVariable(getattr).call_function(
  100. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 636, in call_function
  101. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = handler(tx, *args, **kwargs)
  102. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1167, in call_getattr
  103. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] return obj.var_getattr(tx, name).clone(source=source)
  104. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 218, in var_getattr
  105. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] unimplemented(f"Illegal getattr invocation {name} in strict mode")
  106. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 193, in unimplemented
  107. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] raise Unsupported(msg)
  108. [2023-12-01 09:18:23,264] [2/0] torch._dynamo.variables.higher_order_ops: [ERROR] torch._dynamo.exc.Unsupported: Illegal getattr invocation stride in strict mode
  109. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [WARNING] speculate_subgraph: while introspecting the user-defined autograd.Function, we were unable to trace function `trampoline_autograd_bwd` into a single graph. This means that Dynamo was unable to prove safety for this API and will fall back to eager-mode PyTorch, which could lead to a slowdown.
  110. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] Illegal getattr invocation stride in strict mode
  111. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] Traceback (most recent call last):
  112. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/higher_order_ops.py", line 240, in speculate_subgraph
  113. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] output = f.call_function(tx, args, sub_kwargs)
  114. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  115. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  116. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  117. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  118. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  119. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  120. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  121. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  122. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  123. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  124. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  125. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  126. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  127. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  128. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  129. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  130. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1252, in CALL_FUNCTION_EX
  131. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, argsvars.items, kwargsvars.items)
  132. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  133. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  134. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 643, in call_function
  135. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return self.obj.call_method(tx, self.name, args, kwargs)
  136. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 488, in call_method
  137. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_call(tx, backward, args, kwargs)
  138. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  139. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  140. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  141. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  142. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  143. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  144. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  145. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  146. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  147. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  148. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1264, in CALL_FUNCTION_KW
  149. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, args, kwargs)
  150. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  151. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  152. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  153. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  154. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  155. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  156. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  157. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  158. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  159. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  160. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  161. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  162. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  163. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  164. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  165. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  166. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1303, in LOAD_ATTR
  167. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = BuiltinVariable(getattr).call_function(
  168. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 636, in call_function
  169. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = handler(tx, *args, **kwargs)
  170. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1167, in call_getattr
  171. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] return obj.var_getattr(tx, name).clone(source=source)
  172. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 218, in var_getattr
  173. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] unimplemented(f"Illegal getattr invocation {name} in strict mode")
  174. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 193, in unimplemented
  175. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] raise Unsupported(msg)
  176. [2023-12-01 09:18:26,376] [3/0] torch._dynamo.variables.higher_order_ops: [ERROR] torch._dynamo.exc.Unsupported: Illegal getattr invocation stride in strict mode
  177. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [WARNING] speculate_subgraph: while introspecting the user-defined autograd.Function, we were unable to trace function `trampoline_autograd_bwd` into a single graph. This means that Dynamo was unable to prove safety for this API and will fall back to eager-mode PyTorch, which could lead to a slowdown.
  178. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] Illegal getattr invocation stride in strict mode
  179. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] Traceback (most recent call last):
  180. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/higher_order_ops.py", line 240, in speculate_subgraph
  181. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] output = f.call_function(tx, args, sub_kwargs)
  182. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  183. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  184. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  185. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  186. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  187. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  188. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  189. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  190. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  191. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  192. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  193. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  194. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  195. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  196. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  197. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  198. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1252, in CALL_FUNCTION_EX
  199. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, argsvars.items, kwargsvars.items)
  200. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  201. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  202. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 643, in call_function
  203. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return self.obj.call_method(tx, self.name, args, kwargs)
  204. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 488, in call_method
  205. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_call(tx, backward, args, kwargs)
  206. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  207. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  208. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  209. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  210. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  211. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  212. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  213. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  214. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  215. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  216. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1264, in CALL_FUNCTION_KW
  217. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, args, kwargs)
  218. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  219. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  220. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  221. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  222. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  223. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  224. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  225. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  226. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  227. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  228. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  229. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  230. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  231. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  232. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  233. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  234. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1303, in LOAD_ATTR
  235. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = BuiltinVariable(getattr).call_function(
  236. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 636, in call_function
  237. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = handler(tx, *args, **kwargs)
  238. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1167, in call_getattr
  239. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] return obj.var_getattr(tx, name).clone(source=source)
  240. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 218, in var_getattr
  241. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] unimplemented(f"Illegal getattr invocation {name} in strict mode")
  242. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 193, in unimplemented
  243. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] raise Unsupported(msg)
  244. [2023-12-01 09:18:26,507] [4/0] torch._dynamo.variables.higher_order_ops: [ERROR] torch._dynamo.exc.Unsupported: Illegal getattr invocation stride in strict mode
  245. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [WARNING] speculate_subgraph: while introspecting the user-defined autograd.Function, we were unable to trace function `trampoline_autograd_bwd` into a single graph. This means that Dynamo was unable to prove safety for this API and will fall back to eager-mode PyTorch, which could lead to a slowdown.
  246. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] Illegal getattr invocation stride in strict mode
  247. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] Traceback (most recent call last):
  248. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/higher_order_ops.py", line 240, in speculate_subgraph
  249. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] output = f.call_function(tx, args, sub_kwargs)
  250. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  251. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  252. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  253. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  254. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  255. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  256. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  257. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  258. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  259. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  260. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  261. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  262. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  263. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  264. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  265. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  266. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1252, in CALL_FUNCTION_EX
  267. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, argsvars.items, kwargsvars.items)
  268. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  269. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  270. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 643, in call_function
  271. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return self.obj.call_method(tx, self.name, args, kwargs)
  272. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 488, in call_method
  273. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_call(tx, backward, args, kwargs)
  274. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  275. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  276. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  277. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  278. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  279. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  280. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  281. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  282. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  283. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  284. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1264, in CALL_FUNCTION_KW
  285. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, args, kwargs)
  286. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  287. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  288. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  289. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  290. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  291. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  292. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  293. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  294. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  295. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  296. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  297. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  298. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  299. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  300. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  301. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  302. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1303, in LOAD_ATTR
  303. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = BuiltinVariable(getattr).call_function(
  304. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 636, in call_function
  305. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = handler(tx, *args, **kwargs)
  306. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1167, in call_getattr
  307. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] return obj.var_getattr(tx, name).clone(source=source)
  308. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 218, in var_getattr
  309. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] unimplemented(f"Illegal getattr invocation {name} in strict mode")
  310. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 193, in unimplemented
  311. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] raise Unsupported(msg)
  312. [2023-12-01 09:18:29,655] [5/0] torch._dynamo.variables.higher_order_ops: [ERROR] torch._dynamo.exc.Unsupported: Illegal getattr invocation stride in strict mode
  313. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [WARNING] speculate_subgraph: while introspecting the user-defined autograd.Function, we were unable to trace function `trampoline_autograd_bwd` into a single graph. This means that Dynamo was unable to prove safety for this API and will fall back to eager-mode PyTorch, which could lead to a slowdown.
  314. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] Illegal getattr invocation stride in strict mode
  315. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] Traceback (most recent call last):
  316. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/higher_order_ops.py", line 240, in speculate_subgraph
  317. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] output = f.call_function(tx, args, sub_kwargs)
  318. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  319. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  320. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  321. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  322. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  323. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  324. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  325. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  326. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  327. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  328. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  329. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  330. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  331. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  332. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  333. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  334. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1252, in CALL_FUNCTION_EX
  335. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, argsvars.items, kwargsvars.items)
  336. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  337. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  338. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 643, in call_function
  339. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return self.obj.call_method(tx, self.name, args, kwargs)
  340. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 488, in call_method
  341. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_call(tx, backward, args, kwargs)
  342. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  343. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  344. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  345. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  346. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  347. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  348. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  349. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  350. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 470, in wrapper
  351. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return inner_fn(self, inst)
  352. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1264, in CALL_FUNCTION_KW
  353. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.call_function(fn, args, kwargs)
  354. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 652, in call_function
  355. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] self.push(fn.call_function(self, args, kwargs))
  356. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 248, in call_function
  357. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return super().call_function(tx, args, kwargs)
  358. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 81, in call_function
  359. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return tx.inline_user_function_return(
  360. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in inline_user_function_return
  361. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  362. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2256, in inline_call
  363. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return cls.inline_call_(parent, func, args, kwargs)
  364. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2371, in inline_call_
  365. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] tracer.run()
  366. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  367. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] and self.step()
  368. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  369. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] getattr(self, inst.opname)(inst)
  370. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1303, in LOAD_ATTR
  371. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = BuiltinVariable(getattr).call_function(
  372. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 636, in call_function
  373. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] result = handler(tx, *args, **kwargs)
  374. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1167, in call_getattr
  375. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] return obj.var_getattr(tx, name).clone(source=source)
  376. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 218, in var_getattr
  377. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] unimplemented(f"Illegal getattr invocation {name} in strict mode")
  378. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 193, in unimplemented
  379. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] raise Unsupported(msg)
  380. [2023-12-01 09:18:29,725] [6/0] torch._dynamo.variables.higher_order_ops: [ERROR] torch._dynamo.exc.Unsupported: Illegal getattr invocation stride in strict mode
  381. python3: /tmp/tmpvjczotr9/triton/lib/Conversion/TritonGPUToLLVM/ConvertLayoutOpToLLVM.cpp:973: mlir::Value ConvertLayoutOpConversion::lowerSharedToDotOperandMMA(mlir::triton::gpu::ConvertLayoutOp, ConvertTritonGPUOpToLLVMPattern<mlir::triton::gpu::ConvertLayoutOp>::OpAdaptor, mlir::ConversionPatternRewriter&, const mlir::triton::gpu::MmaEncodingAttr&, const mlir::triton::gpu::DotOperandEncodingAttr&, bool) const: Assertion `false && "Unsupported mma layout found"' failed.
  382. Traceback (most recent call last):
  383. File "/home/robert/languini-kitchen/languini/common_lib/throughput.py", line 132, in <module>
  384. main()
  385. File "/home/robert/languini-kitchen/languini/common_lib/throughput.py", line 82, in main
  386. result = throughput_utils.throughput_test(config=config, model=model)
  387. File "/home/robert/languini-kitchen/languini/common_lib/throughput_utils.py", line 79, in throughput_test
  388. step(inputs, targets, state)
  389. File "/home/robert/languini-kitchen/languini/common_lib/throughput_utils.py", line 65, in step
  390. logits, _ = model(inputs, state)
  391. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
  392. return self._call_impl(*args, **kwargs)
  393. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
  394. return forward_call(*args, **kwargs)
  395. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 489, in _fn
  396. return fn(*args, **kwargs)
  397. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
  398. return self._call_impl(*args, **kwargs)
  399. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
  400. return forward_call(*args, **kwargs)
  401. File "/home/robert/languini-kitchen/languini/projects/moe/model.py", line 91, in forward
  402. x = layer(x, log=log)
  403. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
  404. return self._call_impl(*args, **kwargs)
  405. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
  406. return forward_call(*args, **kwargs)
  407. File "/home/robert/languini-kitchen/languini/projects/moe/lib.py", line 190, in forward
  408. mlp_x = self.mlp(self.ln2(x, log=log), log=log)
  409. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
  410. return self._call_impl(*args, **kwargs)
  411. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
  412. return forward_call(*args, **kwargs)
  413. File "/home/robert/languini-kitchen/languini/projects/moe/lib.py", line 159, in forward
  414. x, reg_loss = self.moe(x)
  415. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
  416. return self._call_impl(*args, **kwargs)
  417. File "/home/robert/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
  418. return forward_call(*args, **kwargs)
  419. File "/home/robert/languini-kitchen/languini/projects/moe/moe_layer/moe_layer_simple.py", line 166, in forward
  420. scores = self.compute_scores(input, sel_indices)
  421. File "/home/robert/languini-kitchen/languini/projects/moe/moe_layer/moe_layer_simple.py", line 97, in compute_scores
  422. scores = cvmm(input, index, self.keys)
  423. File "/home/robert/languini-kitchen/languini/projects/moe/moe_layer/cvmm.py", line 473, in cvmm
  424. return CVMM.apply(x, sel.sel_index, sel.sel, keys, sel.out_index, sel.reduction_weight)
  425. File "/home/robert/.local/lib/python3.10/site-packages/torch/autograd/function.py", line 553, in apply
  426. return super().apply(*args, **kwargs) # type: ignore[misc]
  427. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 655, in catch_errors
  428. return callback(frame, cache_entry, hooks, frame_state)
  429. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 722, in _convert_frame
  430. result = inner_convert(frame, cache_entry, hooks, frame_state)
  431. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 383, in _convert_frame_assert
  432. compiled_product = _compile(
  433. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 646, in _compile
  434. guarded_code = compile_inner(code, one_graph, hooks, transform)
  435. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
  436. r = func(*args, **kwargs)
  437. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in compile_inner
  438. out_code = transform_code_object(code, transform)
  439. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1033, in transform_code_object
  440. transformations(instructions, code_options)
  441. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 151, in _fn
  442. return fn(*args, **kwargs)
  443. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 527, in transform
  444. tracer.run()
  445. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2123, in run
  446. super().run()
  447. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 818, in run
  448. and self.step()
  449. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in step
  450. getattr(self, inst.opname)(inst)
  451. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2238, in RETURN_VALUE
  452. self.output.compile_subgraph(
  453. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 938, in compile_subgraph
  454. self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
  455. File "/usr/lib/python3.10/contextlib.py", line 79, in inner
  456. return func(*args, **kwds)
  457. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1080, in compile_and_call_fx_graph
  458. compiled_fn = self.call_user_compiler(gm)
  459. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
  460. r = func(*args, **kwargs)
  461. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1152, in call_user_compiler
  462. raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
  463. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1133, in call_user_compiler
  464. compiled_fn = compiler_fn(gm, self.example_inputs())
  465. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
  466. compiled_gm = compiler_fn(gm, example_inputs)
  467. File "/home/robert/.local/lib/python3.10/site-packages/torch/__init__.py", line 1657, in __call__
  468. return compile_fx(model_, inputs_, config_patches=self.config)
  469. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1168, in compile_fx
  470. return aot_autograd(
  471. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
  472. cg = aot_module_simplified(gm, example_inputs, **kwargs)
  473. File "/home/robert/.local/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 885, in aot_module_simplified
  474. compiled_fn = create_aot_dispatcher_function(
  475. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
  476. r = func(*args, **kwargs)
  477. File "/home/robert/.local/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 598, in create_aot_dispatcher_function
  478. compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
  479. File "/home/robert/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 424, in aot_wrapper_dedupe
  480. return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
  481. File "/home/robert/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 629, in aot_wrapper_synthetic_base
  482. return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
  483. File "/home/robert/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 97, in aot_dispatch_base
  484. compiled_fw = compiler(fw_module, updated_flat_args)
  485. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
  486. r = func(*args, **kwargs)
  487. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1100, in fw_compiler_base
  488. return inner_compile(
  489. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 83, in debug_wrapper
  490. inner_compiled_fn = compiler_fn(gm, example_inputs)
  491. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/debug.py", line 303, in inner
  492. return fn(*args, **kwargs)
  493. File "/usr/lib/python3.10/contextlib.py", line 79, in inner
  494. return func(*args, **kwds)
  495. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 320, in compile_fx_inner
  496. compiled_graph = fx_codegen_and_compile(
  497. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 550, in fx_codegen_and_compile
  498. compiled_fn = graph.compile_to_fn()
  499. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/graph.py", line 1115, in compile_to_fn
  500. return self.compile_to_module().call
  501. File "/home/robert/.local/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 244, in time_wrapper
  502. r = func(*args, **kwargs)
  503. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/graph.py", line 1069, in compile_to_module
  504. mod = PyCodeCache.load_by_key_path(
  505. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1886, in load_by_key_path
  506. exec(code, mod.__dict__, mod.__dict__)
  507. File "/tmp/torchinductor_robert/2g/c2g6u7gsiwum72oblqalbmuz3xxcpsqnol4mu73pkbqfnqhu7anw.py", line 303, in <module>
  508. async_compile.wait(globals())
  509. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 2465, in wait
  510. scope[key] = result.result()
  511. File "/home/robert/.local/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 2308, in result
  512. self.future.result()
  513. File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
  514. return self.__get_result()
  515. File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
  516. raise self._exception
  517. torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
  518. BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
  519.  
  520. Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
  521.  
  522.  
  523. You can suppress this exception and fall back to eager by setting:
  524. import torch._dynamo
  525. torch._dynamo.config.suppress_errors = True
  526.  
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!