API tools faq
paste
A-Vladimir

Log 18

A-Vladimir
Jul 25th, 2025
82
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 16.30 KB | None | 0 0
raw download clone embed print report
  1.  
  2. Starting up...
  3. Preparing first run of this install - may take a minute or two...
  4. Started Invoke process with PID: 53375
  5. bitsandbytes library load error: Configured CUDA binary not found at /mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/bitsandbytes/libbitsandbytes_rocm63.so
  6. Traceback (most recent call last):
  7. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/bitsandbytes/cextension.py", line 290, in <module>
  8. lib = get_native_library()
  9. ^^^^^^^^^^^^^^^^^^^^
  10. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/bitsandbytes/cextension.py", line 270, in get_native_library
  11. raise RuntimeError(f"Configured CUDA binary not found at {cuda_binary_path}")
  12. RuntimeError: Configured CUDA binary not found at /mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/bitsandbytes/libbitsandbytes_rocm63.so
  13. [2025-07-25 16:29:43,647]::[InvokeAI]::INFO --> Using torch device: Radeon RX 7900 XTX
  14. [2025-07-25 16:29:43,656]::[InvokeAI]::INFO --> cuDNN version: 3003000
  15. >> patchmatch.patch_match: INFO - Compiling and loading c extensions from "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/patchmatch".
  16. [2025-07-25 16:29:46,396]::[InvokeAI]::INFO --> Patchmatch initialized
  17. [2025-07-25 16:29:47,443]::[InvokeAI]::INFO --> InvokeAI version 6.2.0
  18. [2025-07-25 16:29:47,443]::[InvokeAI]::INFO --> Root directory = /mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke
  19. [2025-07-25 16:29:47,443]::[InvokeAI]::INFO --> Initializing database at /mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/databases/invokeai.db
  20. [2025-07-25 16:29:47,448]::[ModelManagerService]::INFO --> [MODEL CACHE] Calculated model RAM cache size: 21488.00 MB. Heuristics applied: [1, 2].
  21. [2025-07-25 16:29:47,941]::[InvokeAI]::INFO --> Invoke running on http://127.0.0.1:9090 (Press CTRL+C to quit)
  22. [2025-07-25 16:29:53,822]::[InvokeAI]::INFO --> Executing queue item 2300, session 560e9f9a-1152-4645-ad0e-ece820da537c
  23. [2025-07-25 16:30:00,086]::[ModelManagerService]::INFO --> [MODEL CACHE] Loaded model '7645d695-637e-43bc-9f63-9207ba827f49:text_encoder_2' (T5EncoderModel) onto cuda device in 5.94s. Total model size: 4922.14MB, VRAM: 4922.14MB (100.0%)
  24. You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers
  25. [2025-07-25 16:30:00,222]::[ModelManagerService]::INFO --> [MODEL CACHE] Loaded model '7645d695-637e-43bc-9f63-9207ba827f49:tokenizer_2' (T5TokenizerFast) onto cuda device in 0.00s. Total model size: 0.03MB, VRAM: 0.00MB (0.0%)
  26. /mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/bitsandbytes/autograd/_functions.py:185: UserWarning: MatMul8bitLt: inputs will be cast from torch.bfloat16 to float16 during quantization
  27. warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
  28. [2025-07-25 16:30:00,359]::[InvokeAI]::ERROR --> Error while invoking session 560e9f9a-1152-4645-ad0e-ece820da537c, invocation f4bcd14b-b76a-4974-bc06-9db8eff73e50 (flux_text_encoder):
  29. 🚨 CUDA VERSION MISMATCH 🚨
  30. Requested CUDA version: 6.3
  31. Detected PyTorch CUDA version: 6.3
  32. Available pre-compiled versions:
  33. - 11.8
  34. - 12.0
  35. - 12.1
  36. - 12.2
  37. - 12.3
  38. - 12.4
  39. - 12.5
  40. - 12.6
  41. - 12.8
  42. - 12.9
  43. This means:
  44. The version you're trying to use is NOT distributed with this package
  45. Attempted to use bitsandbytes native library functionality but it's not available.
  46. This typically happens when:
  47. 1. bitsandbytes doesn't ship with a pre-compiled binary for your CUDA version
  48. 2. The library wasn't compiled properly during installation from source
  49. To make bitsandbytes work, the compiled library version MUST exactly match the linked CUDA version.
  50. If your CUDA version doesn't have a pre-compiled binary, you MUST compile from source.
  51. You have two options:
  52. 1. COMPILE FROM SOURCE (required if no binary exists):
  53. https://huggingface.co/docs/bitsandbytes/main/en/installation#cuda-compile
  54. 2. Use BNB_CUDA_VERSION to specify a DIFFERENT CUDA version from the detected one, which is installed on your machine and matching an available pre-compiled version listed above
  55. Original error: Configured CUDA binary not found at /mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/bitsandbytes/libbitsandbytes_rocm63.so
  56. 🔍 Run this command for detailed diagnostics:
  57. python -m bitsandbytes
  58. If you've tried everything and still have issues:
  59. 1. Include ALL version info (operating system, bitsandbytes, pytorch, cuda, python)
  60. 2. Describe what you've tried in detail
  61. 3. Open an issue with this information:
  62. https://github.com/bitsandbytes-foundation/bitsandbytes/issues
  63. Native code method attempted to call: lib.cint8_vector_quant()
  64. [2025-07-25 16:30:00,360]::[InvokeAI]::ERROR --> Traceback (most recent call last):
  65. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/invokeai/app/services/session_processor/session_processor_default.py", line 130, in run_node
  66. output = invocation.invoke_internal(context=context, services=self._services)
  67. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  68. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/invokeai/app/invocations/baseinvocation.py", line 241, in invoke_internal
  69. output = self.invoke(context)
  70. ^^^^^^^^^^^^^^^^^^^^
  71. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
  72. return func(*args, **kwargs)
  73. ^^^^^^^^^^^^^^^^^^^^^
  74. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/invokeai/app/invocations/flux_text_encoder.py", line 59, in invoke
  75. t5_embeddings = self._t5_encode(context)
  76. ^^^^^^^^^^^^^^^^^^^^^^^^
  77. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/invokeai/app/invocations/flux_text_encoder.py", line 118, in _t5_encode
  78. prompt_embeds = t5_encoder(prompt)
  79. ^^^^^^^^^^^^^^^^^^
  80. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
  81. return self._call_impl(*args, **kwargs)
  82. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  83. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
  84. return forward_call(*args, **kwargs)
  85. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  86. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/invokeai/backend/flux/modules/conditioner.py", line 36, in forward
  87. outputs = self.hf_module(
  88. ^^^^^^^^^^^^^^^
  89. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
  90. return self._call_impl(*args, **kwargs)
  91. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  92. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
  93. return forward_call(*args, **kwargs)
  94. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  95. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/transformers/models/t5/modeling_t5.py", line 1990, in forward
  96. encoder_outputs = self.encoder(
  97. ^^^^^^^^^^^^^
  98. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
  99. return self._call_impl(*args, **kwargs)
  100. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  101. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
  102. return forward_call(*args, **kwargs)
  103. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  104. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/transformers/models/t5/modeling_t5.py", line 1105, in forward
  105. layer_outputs = layer_module(
  106. ^^^^^^^^^^^^^
  107. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/transformers/modeling_layers.py", line 83, in __call__
  108. return super().__call__(*args, **kwargs)
  109. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  110. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
  111. return self._call_impl(*args, **kwargs)
  112. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  113. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
  114. return forward_call(*args, **kwargs)
  115. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  116. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/transformers/models/t5/modeling_t5.py", line 678, in forward
  117. self_attention_outputs = self.layer[0](
  118. ^^^^^^^^^^^^^^
  119. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
  120. return self._call_impl(*args, **kwargs)
  121. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  122. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
  123. return forward_call(*args, **kwargs)
  124. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  125. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/transformers/models/t5/modeling_t5.py", line 596, in forward
  126. attention_output = self.SelfAttention(
  127. ^^^^^^^^^^^^^^^^^^^
  128. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
  129. return self._call_impl(*args, **kwargs)
  130. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  131. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
  132. return forward_call(*args, **kwargs)
  133. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  134. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
  135. query_states = self.q(hidden_states)
  136. ^^^^^^^^^^^^^^^^^^^^^
  137. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
  138. return self._call_impl(*args, **kwargs)
  139. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  140. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
  141. return forward_call(*args, **kwargs)
  142. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  143. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/invokeai/backend/model_manager/load/model_cache/torch_module_autocast/custom_modules/custom_invoke_linear_8_bit_lt.py", line 44, in forward
  144. return super().forward(x)
  145. ^^^^^^^^^^^^^^^^^^
  146. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/invokeai/backend/quantization/bnb_llm_int8.py", line 119, in forward
  147. return bnb.matmul(x, self.weight, bias=self.bias, state=matmul_state)
  148. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  149. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/bitsandbytes/autograd/_functions.py", line 369, in matmul
  150. return MatMul8bitLt.apply(A, B, out, bias, state)
  151. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  152. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/autograd/function.py", line 575, in apply
  153. return super().apply(*args, **kwargs) # type: ignore[misc]
  154. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  155. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/bitsandbytes/autograd/_functions.py", line 196, in forward
  156. CA, SCA, outlier_cols = F.int8_vectorwise_quant(A.to(torch.float16), threshold=state.threshold)
  157. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  158. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/bitsandbytes/functional.py", line 2245, in int8_vectorwise_quant
  159. return torch.ops.bitsandbytes.int8_vectorwise_quant.default(A, threshold)
  160. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  161. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/_ops.py", line 756, in __call__
  162. return self._op(*args, **kwargs)
  163. ^^^^^^^^^^^^^^^^^^^^^^^^^
  164. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/_compile.py", line 51, in inner
  165. return disable_fn(*args, **kwargs)
  166. ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  167. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
  168. return fn(*args, **kwargs)
  169. ^^^^^^^^^^^^^^^^^^^
  170. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/torch/library.py", line 719, in func_no_dynamo
  171. return func(*args, **kwargs)
  172. ^^^^^^^^^^^^^^^^^^^^^
  173. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/bitsandbytes/backends/cuda/ops.py", line 151, in _
  174. lib.cint8_vector_quant(
  175. File "/mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/bitsandbytes/cextension.py", line 251, in throw_on_call
  176. raise RuntimeError(f"{self.formatted_error}Native code method attempted to call: lib.{name}()")
  177. RuntimeError:
  178. 🚨 CUDA VERSION MISMATCH 🚨
  179. Requested CUDA version: 6.3
  180. Detected PyTorch CUDA version: 6.3
  181. Available pre-compiled versions:
  182. - 11.8
  183. - 12.0
  184. - 12.1
  185. - 12.2
  186. - 12.3
  187. - 12.4
  188. - 12.5
  189. - 12.6
  190. - 12.8
  191. - 12.9
  192. This means:
  193. The version you're trying to use is NOT distributed with this package
  194. Attempted to use bitsandbytes native library functionality but it's not available.
  195. This typically happens when:
  196. 1. bitsandbytes doesn't ship with a pre-compiled binary for your CUDA version
  197. 2. The library wasn't compiled properly during installation from source
  198. To make bitsandbytes work, the compiled library version MUST exactly match the linked CUDA version.
  199. If your CUDA version doesn't have a pre-compiled binary, you MUST compile from source.
  200. You have two options:
  201. 1. COMPILE FROM SOURCE (required if no binary exists):
  202. https://huggingface.co/docs/bitsandbytes/main/en/installation#cuda-compile
  203. 2. Use BNB_CUDA_VERSION to specify a DIFFERENT CUDA version from the detected one, which is installed on your machine and matching an available pre-compiled version listed above
  204. Original error: Configured CUDA binary not found at /mnt/5868f70d-4271-4fe4-a06e-5d1455ac9dc7/Invoke/.venv/lib/python3.12/site-packages/bitsandbytes/libbitsandbytes_rocm63.so
  205. 🔍 Run this command for detailed diagnostics:
  206. python -m bitsandbytes
  207. If you've tried everything and still have issues:
  208. 1. Include ALL version info (operating system, bitsandbytes, pytorch, cuda, python)
  209. 2. Describe what you've tried in detail
  210. 3. Open an issue with this information:
  211. https://github.com/bitsandbytes-foundation/bitsandbytes/issues
  212. Native code method attempted to call: lib.cint8_vector_quant()
  213. [2025-07-25 16:30:00,384]::[InvokeAI]::INFO --> Graph stats: 560e9f9a-1152-4645-ad0e-ece820da537c
  214. Node Calls Seconds VRAM Used
  215. flux_model_loader 1 0.078s 0.000G
  216. string 1 0.006s 0.000G
  217. flux_text_encoder 1 6.401s 4.809G
  218. TOTAL GRAPH EXECUTION TIME: 6.485s
  219. TOTAL GRAPH WALL TIME: 6.528s
  220. RAM used by InvokeAI process: 6.50G (+4.938G)
  221. RAM used to load models: 4.81G
  222. VRAM in use: 4.563G
  223. RAM cache statistics:
  224. Model cache hits: 2
  225. Model cache misses: 2
  226. Models cached: 2
  227. Models cleared from cache: 0
  228. Cache high water mark: 4.81/0.00G
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!