API tools faq
paste
Guest User

Untitled

a guest
May 15th, 2025
16
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 76.08 KB | None | 0 0
raw download clone embed print report
  1. ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
  2. ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: yes
  3. ggml_cuda_init: found 2 CUDA devices:
  4. Device 0: Tesla P100-PCIE-16GB, compute capability 6.0, VMM: yes
  5. Device 1: Tesla P100-PCIE-16GB, compute capability 6.0, VMM: yes
  6. | model | size | params | backend | ngl | threads | fa | fmoe | test | t/s |
  7. | ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -: | ---: | ------------: | ---------------: |
  8. llama_model_loader: additional 2 GGUFs metadata loaded.
  9. llama_model_loader: loaded meta data with 40 key-value pairs and 1131 tensors from gguf/moe/Qwen3-235B-A22B-mix-IQ3_K-00001-of-00003.gguf (version GGUF V3 (latest))
  10. llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
  11. llama_model_loader: - kv 0: general.architecture str = qwen3moe
  12. llama_model_loader: - kv 1: general.type str = model
  13. llama_model_loader: - kv 2: general.name str = Qwen3 235B A22B
  14. llama_model_loader: - kv 3: general.basename str = Qwen3
  15. llama_model_loader: - kv 4: general.size_label str = 235B-A22B
  16. llama_model_loader: - kv 5: general.license str = apache-2.0
  17. llama_model_loader: - kv 6: general.license.link str = https://huggingface.co/Qwen/Qwen3-235...
  18. llama_model_loader: - kv 7: general.tags arr[str,1] = ["text-generation"]
  19. llama_model_loader: - kv 8: qwen3moe.block_count u32 = 94
  20. llama_model_loader: - kv 9: qwen3moe.context_length u32 = 40960
  21. llama_model_loader: - kv 10: qwen3moe.embedding_length u32 = 4096
  22. llama_model_loader: - kv 11: qwen3moe.feed_forward_length u32 = 12288
  23. llama_model_loader: - kv 12: qwen3moe.attention.head_count u32 = 64
  24. llama_model_loader: - kv 13: qwen3moe.attention.head_count_kv u32 = 4
  25. llama_model_loader: - kv 14: qwen3moe.rope.freq_base f32 = 1000000.000000
  26. llama_model_loader: - kv 15: qwen3moe.attention.layer_norm_rms_epsilon f32 = 0.000001
  27. llama_model_loader: - kv 16: qwen3moe.expert_used_count u32 = 8
  28. llama_model_loader: - kv 17: qwen3moe.attention.key_length u32 = 128
  29. llama_model_loader: - kv 18: qwen3moe.attention.value_length u32 = 128
  30. llama_model_loader: - kv 19: general.file_type u32 = 139
  31. llama_model_loader: - kv 20: qwen3moe.expert_count u32 = 128
  32. llama_model_loader: - kv 21: qwen3moe.expert_feed_forward_length u32 = 1536
  33. llama_model_loader: - kv 22: general.quantization_version u32 = 2
  34. llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2
  35. llama_model_loader: - kv 24: tokenizer.ggml.pre str = qwen2
  36. llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,151936] = ["!", "\"", "#", "$", "%", "&", "'", ...
  37. llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,151936] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
  38. llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
  39. llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 151645
  40. llama_model_loader: - kv 29: tokenizer.ggml.padding_token_id u32 = 151643
  41. llama_model_loader: - kv 30: tokenizer.ggml.bos_token_id u32 = 151643
  42. llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = false
  43. llama_model_loader: - kv 32: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>...
  44. llama_model_loader: - kv 33: quantize.imatrix.file str = /mnt/raid/models/ubergarm/Qwen3-235B-...
  45. llama_model_loader: - kv 34: quantize.imatrix.dataset str = calibration_data_v5_rc.txt
  46. llama_model_loader: - kv 35: quantize.imatrix.entries_count i32 = 753
  47. llama_model_loader: - kv 36: quantize.imatrix.chunks_count i32 = 225
  48. llama_model_loader: - kv 37: split.no u16 = 0
  49. llama_model_loader: - kv 38: split.count u16 = 3
  50. llama_model_loader: - kv 39: split.tensors.count i32 = 1131
  51. llama_model_loader: - type f32: 471 tensors
  52. llama_model_loader: - type q8_0: 2 tensors
  53. llama_model_loader: - type iq3_k: 188 tensors
  54. llama_model_loader: - type iq4_k: 94 tensors
  55. llama_model_loader: - type iq6_k: 376 tensors
  56. llm_load_vocab: special tokens cache size = 26
  57. llm_load_vocab: token to piece cache size = 0.9311 MB
  58. llm_load_print_meta: format = GGUF V3 (latest)
  59. llm_load_print_meta: arch = qwen3moe
  60. llm_load_print_meta: vocab type = BPE
  61. llm_load_print_meta: n_vocab = 151936
  62. llm_load_print_meta: n_merges = 151387
  63. llm_load_print_meta: vocab_only = 0
  64. llm_load_print_meta: n_ctx_train = 40960
  65. llm_load_print_meta: n_embd = 4096
  66. llm_load_print_meta: n_layer = 94
  67. llm_load_print_meta: n_head = 64
  68. llm_load_print_meta: n_head_kv = 4
  69. llm_load_print_meta: n_rot = 128
  70. llm_load_print_meta: n_swa = 0
  71. llm_load_print_meta: n_swa_pattern = 1
  72. llm_load_print_meta: n_embd_head_k = 128
  73. llm_load_print_meta: n_embd_head_v = 128
  74. llm_load_print_meta: n_gqa = 16
  75. llm_load_print_meta: n_embd_k_gqa = 512
  76. llm_load_print_meta: n_embd_v_gqa = 512
  77. llm_load_print_meta: f_norm_eps = 0.0e+00
  78. llm_load_print_meta: f_norm_rms_eps = 1.0e-06
  79. llm_load_print_meta: f_clamp_kqv = 0.0e+00
  80. llm_load_print_meta: f_max_alibi_bias = 0.0e+00
  81. llm_load_print_meta: f_logit_scale = 0.0e+00
  82. llm_load_print_meta: n_ff = 12288
  83. llm_load_print_meta: n_expert = 128
  84. llm_load_print_meta: n_expert_used = 8
  85. llm_load_print_meta: causal attn = 1
  86. llm_load_print_meta: pooling type = 0
  87. llm_load_print_meta: rope type = 2
  88. llm_load_print_meta: rope scaling = linear
  89. llm_load_print_meta: freq_base_train = 1000000.0
  90. llm_load_print_meta: freq_scale_train = 1
  91. llm_load_print_meta: n_ctx_orig_yarn = 40960
  92. llm_load_print_meta: rope_finetuned = unknown
  93. llm_load_print_meta: ssm_d_conv = 0
  94. llm_load_print_meta: ssm_d_inner = 0
  95. llm_load_print_meta: ssm_d_state = 0
  96. llm_load_print_meta: ssm_dt_rank = 0
  97. llm_load_print_meta: model type = ?B
  98. llm_load_print_meta: model ftype = IQ3_K - 3.4325 bpw
  99. llm_load_print_meta: model params = 235.094 B
  100. llm_load_print_meta: model size = 106.830 GiB (3.903 BPW)
  101. llm_load_print_meta: repeating layers = 105.598 GiB (3.879 BPW, 233.849 B parameters)
  102. llm_load_print_meta: general.name = Qwen3 235B A22B
  103. llm_load_print_meta: BOS token = 151643 '<|endoftext|>'
  104. llm_load_print_meta: EOS token = 151645 '<|im_end|>'
  105. llm_load_print_meta: PAD token = 151643 '<|endoftext|>'
  106. llm_load_print_meta: LF token = 148848 'ÄĬ'
  107. llm_load_print_meta: EOT token = 151645 '<|im_end|>'
  108. llm_load_print_meta: max token length = 256
  109. llm_load_print_meta: n_ff_exp = 1536
  110. llm_load_tensors: ggml ctx size = 1.49 MiB
  111. Tensor blk.0.ffn_gate_exps.weight buffer type overriden to CUDA0
  112. Tensor blk.0.ffn_down_exps.weight buffer type overriden to CUDA0
  113. Tensor blk.0.ffn_up_exps.weight buffer type overriden to CUDA0
  114. Tensor blk.1.ffn_gate_exps.weight buffer type overriden to CUDA0
  115. Tensor blk.1.ffn_down_exps.weight buffer type overriden to CUDA0
  116. Tensor blk.1.ffn_up_exps.weight buffer type overriden to CUDA0
  117. Tensor blk.2.ffn_gate_exps.weight buffer type overriden to CUDA0
  118. Tensor blk.2.ffn_down_exps.weight buffer type overriden to CUDA0
  119. Tensor blk.2.ffn_up_exps.weight buffer type overriden to CUDA0
  120. Tensor blk.3.ffn_gate_exps.weight buffer type overriden to CUDA0
  121. Tensor blk.3.ffn_down_exps.weight buffer type overriden to CUDA0
  122. Tensor blk.3.ffn_up_exps.weight buffer type overriden to CUDA0
  123. Tensor blk.4.ffn_gate_exps.weight buffer type overriden to CUDA0
  124. Tensor blk.4.ffn_down_exps.weight buffer type overriden to CUDA0
  125. Tensor blk.4.ffn_up_exps.weight buffer type overriden to CUDA0
  126. Tensor blk.5.ffn_gate_exps.weight buffer type overriden to CUDA0
  127. Tensor blk.5.ffn_down_exps.weight buffer type overriden to CUDA0
  128. Tensor blk.5.ffn_up_exps.weight buffer type overriden to CUDA0
  129. Tensor blk.6.ffn_gate_exps.weight buffer type overriden to CUDA0
  130. Tensor blk.6.ffn_down_exps.weight buffer type overriden to CUDA0
  131. Tensor blk.6.ffn_up_exps.weight buffer type overriden to CUDA0
  132. Tensor blk.7.ffn_gate_exps.weight buffer type overriden to CUDA0
  133. Tensor blk.7.ffn_down_exps.weight buffer type overriden to CUDA0
  134. Tensor blk.7.ffn_up_exps.weight buffer type overriden to CUDA0
  135. Tensor blk.8.ffn_gate_exps.weight buffer type overriden to CUDA0
  136. Tensor blk.8.ffn_down_exps.weight buffer type overriden to CUDA0
  137. Tensor blk.8.ffn_up_exps.weight buffer type overriden to CUDA0
  138. Tensor blk.9.ffn_gate_exps.weight buffer type overriden to CUDA0
  139. Tensor blk.9.ffn_down_exps.weight buffer type overriden to CUDA0
  140. Tensor blk.9.ffn_up_exps.weight buffer type overriden to CUDA0
  141. Tensor blk.10.attn_norm.weight buffer type overriden to CUDA0
  142. Tensor blk.10.attn_q.weight buffer type overriden to CUDA0
  143. Tensor blk.10.attn_k.weight buffer type overriden to CUDA0
  144. Tensor blk.10.attn_v.weight buffer type overriden to CUDA0
  145. Tensor blk.10.attn_output.weight buffer type overriden to CUDA0
  146. Tensor blk.10.attn_k_norm.weight buffer type overriden to CUDA0
  147. Tensor blk.10.attn_q_norm.weight buffer type overriden to CUDA0
  148. Tensor blk.10.ffn_norm.weight buffer type overriden to CUDA0
  149. Tensor blk.10.ffn_gate_inp.weight buffer type overriden to CUDA0
  150. Tensor blk.10.ffn_gate_exps.weight buffer type overriden to CPU
  151. Tensor blk.10.ffn_down_exps.weight buffer type overriden to CPU
  152. Tensor blk.10.ffn_up_exps.weight buffer type overriden to CPU
  153. Tensor blk.11.attn_norm.weight buffer type overriden to CUDA0
  154. Tensor blk.11.attn_q.weight buffer type overriden to CUDA0
  155. Tensor blk.11.attn_k.weight buffer type overriden to CUDA0
  156. Tensor blk.11.attn_v.weight buffer type overriden to CUDA0
  157. Tensor blk.11.attn_output.weight buffer type overriden to CUDA0
  158. Tensor blk.11.attn_k_norm.weight buffer type overriden to CUDA0
  159. Tensor blk.11.attn_q_norm.weight buffer type overriden to CUDA0
  160. Tensor blk.11.ffn_norm.weight buffer type overriden to CUDA0
  161. Tensor blk.11.ffn_gate_inp.weight buffer type overriden to CUDA0
  162. Tensor blk.11.ffn_gate_exps.weight buffer type overriden to CPU
  163. Tensor blk.11.ffn_down_exps.weight buffer type overriden to CPU
  164. Tensor blk.11.ffn_up_exps.weight buffer type overriden to CPU
  165. Tensor blk.12.attn_norm.weight buffer type overriden to CUDA0
  166. Tensor blk.12.attn_q.weight buffer type overriden to CUDA0
  167. Tensor blk.12.attn_k.weight buffer type overriden to CUDA0
  168. Tensor blk.12.attn_v.weight buffer type overriden to CUDA0
  169. Tensor blk.12.attn_output.weight buffer type overriden to CUDA0
  170. Tensor blk.12.attn_k_norm.weight buffer type overriden to CUDA0
  171. Tensor blk.12.attn_q_norm.weight buffer type overriden to CUDA0
  172. Tensor blk.12.ffn_norm.weight buffer type overriden to CUDA0
  173. Tensor blk.12.ffn_gate_inp.weight buffer type overriden to CUDA0
  174. Tensor blk.12.ffn_gate_exps.weight buffer type overriden to CPU
  175. Tensor blk.12.ffn_down_exps.weight buffer type overriden to CPU
  176. Tensor blk.12.ffn_up_exps.weight buffer type overriden to CPU
  177. Tensor blk.13.attn_norm.weight buffer type overriden to CUDA0
  178. Tensor blk.13.attn_q.weight buffer type overriden to CUDA0
  179. Tensor blk.13.attn_k.weight buffer type overriden to CUDA0
  180. Tensor blk.13.attn_v.weight buffer type overriden to CUDA0
  181. Tensor blk.13.attn_output.weight buffer type overriden to CUDA0
  182. Tensor blk.13.attn_k_norm.weight buffer type overriden to CUDA0
  183. Tensor blk.13.attn_q_norm.weight buffer type overriden to CUDA0
  184. Tensor blk.13.ffn_norm.weight buffer type overriden to CUDA0
  185. Tensor blk.13.ffn_gate_inp.weight buffer type overriden to CUDA0
  186. Tensor blk.13.ffn_gate_exps.weight buffer type overriden to CPU
  187. Tensor blk.13.ffn_down_exps.weight buffer type overriden to CPU
  188. Tensor blk.13.ffn_up_exps.weight buffer type overriden to CPU
  189. Tensor blk.14.attn_norm.weight buffer type overriden to CUDA0
  190. Tensor blk.14.attn_q.weight buffer type overriden to CUDA0
  191. Tensor blk.14.attn_k.weight buffer type overriden to CUDA0
  192. Tensor blk.14.attn_v.weight buffer type overriden to CUDA0
  193. Tensor blk.14.attn_output.weight buffer type overriden to CUDA0
  194. Tensor blk.14.attn_k_norm.weight buffer type overriden to CUDA0
  195. Tensor blk.14.attn_q_norm.weight buffer type overriden to CUDA0
  196. Tensor blk.14.ffn_norm.weight buffer type overriden to CUDA0
  197. Tensor blk.14.ffn_gate_inp.weight buffer type overriden to CUDA0
  198. Tensor blk.14.ffn_gate_exps.weight buffer type overriden to CPU
  199. Tensor blk.14.ffn_down_exps.weight buffer type overriden to CPU
  200. Tensor blk.14.ffn_up_exps.weight buffer type overriden to CPU
  201. Tensor blk.15.attn_norm.weight buffer type overriden to CUDA0
  202. Tensor blk.15.attn_q.weight buffer type overriden to CUDA0
  203. Tensor blk.15.attn_k.weight buffer type overriden to CUDA0
  204. Tensor blk.15.attn_v.weight buffer type overriden to CUDA0
  205. Tensor blk.15.attn_output.weight buffer type overriden to CUDA0
  206. Tensor blk.15.attn_k_norm.weight buffer type overriden to CUDA0
  207. Tensor blk.15.attn_q_norm.weight buffer type overriden to CUDA0
  208. Tensor blk.15.ffn_norm.weight buffer type overriden to CUDA0
  209. Tensor blk.15.ffn_gate_inp.weight buffer type overriden to CUDA0
  210. Tensor blk.15.ffn_gate_exps.weight buffer type overriden to CPU
  211. Tensor blk.15.ffn_down_exps.weight buffer type overriden to CPU
  212. Tensor blk.15.ffn_up_exps.weight buffer type overriden to CPU
  213. Tensor blk.16.attn_norm.weight buffer type overriden to CUDA0
  214. Tensor blk.16.attn_q.weight buffer type overriden to CUDA0
  215. Tensor blk.16.attn_k.weight buffer type overriden to CUDA0
  216. Tensor blk.16.attn_v.weight buffer type overriden to CUDA0
  217. Tensor blk.16.attn_output.weight buffer type overriden to CUDA0
  218. Tensor blk.16.attn_k_norm.weight buffer type overriden to CUDA0
  219. Tensor blk.16.attn_q_norm.weight buffer type overriden to CUDA0
  220. Tensor blk.16.ffn_norm.weight buffer type overriden to CUDA0
  221. Tensor blk.16.ffn_gate_inp.weight buffer type overriden to CUDA0
  222. Tensor blk.16.ffn_gate_exps.weight buffer type overriden to CPU
  223. Tensor blk.16.ffn_down_exps.weight buffer type overriden to CPU
  224. Tensor blk.16.ffn_up_exps.weight buffer type overriden to CPU
  225. Tensor blk.17.attn_norm.weight buffer type overriden to CUDA0
  226. Tensor blk.17.attn_q.weight buffer type overriden to CUDA0
  227. Tensor blk.17.attn_k.weight buffer type overriden to CUDA0
  228. Tensor blk.17.attn_v.weight buffer type overriden to CUDA0
  229. Tensor blk.17.attn_output.weight buffer type overriden to CUDA0
  230. Tensor blk.17.attn_k_norm.weight buffer type overriden to CUDA0
  231. Tensor blk.17.attn_q_norm.weight buffer type overriden to CUDA0
  232. Tensor blk.17.ffn_norm.weight buffer type overriden to CUDA0
  233. Tensor blk.17.ffn_gate_inp.weight buffer type overriden to CUDA0
  234. Tensor blk.17.ffn_gate_exps.weight buffer type overriden to CPU
  235. Tensor blk.17.ffn_down_exps.weight buffer type overriden to CPU
  236. Tensor blk.17.ffn_up_exps.weight buffer type overriden to CPU
  237. Tensor blk.18.attn_norm.weight buffer type overriden to CUDA0
  238. Tensor blk.18.attn_q.weight buffer type overriden to CUDA0
  239. Tensor blk.18.attn_k.weight buffer type overriden to CUDA0
  240. Tensor blk.18.attn_v.weight buffer type overriden to CUDA0
  241. Tensor blk.18.attn_output.weight buffer type overriden to CUDA0
  242. Tensor blk.18.attn_k_norm.weight buffer type overriden to CUDA0
  243. Tensor blk.18.attn_q_norm.weight buffer type overriden to CUDA0
  244. Tensor blk.18.ffn_norm.weight buffer type overriden to CUDA0
  245. Tensor blk.18.ffn_gate_inp.weight buffer type overriden to CUDA0
  246. Tensor blk.18.ffn_gate_exps.weight buffer type overriden to CPU
  247. Tensor blk.18.ffn_down_exps.weight buffer type overriden to CPU
  248. Tensor blk.18.ffn_up_exps.weight buffer type overriden to CPU
  249. Tensor blk.19.attn_norm.weight buffer type overriden to CUDA0
  250. Tensor blk.19.attn_q.weight buffer type overriden to CUDA0
  251. Tensor blk.19.attn_k.weight buffer type overriden to CUDA0
  252. Tensor blk.19.attn_v.weight buffer type overriden to CUDA0
  253. Tensor blk.19.attn_output.weight buffer type overriden to CUDA0
  254. Tensor blk.19.attn_k_norm.weight buffer type overriden to CUDA0
  255. Tensor blk.19.attn_q_norm.weight buffer type overriden to CUDA0
  256. Tensor blk.19.ffn_norm.weight buffer type overriden to CUDA0
  257. Tensor blk.19.ffn_gate_inp.weight buffer type overriden to CUDA0
  258. Tensor blk.19.ffn_gate_exps.weight buffer type overriden to CPU
  259. Tensor blk.19.ffn_down_exps.weight buffer type overriden to CPU
  260. Tensor blk.19.ffn_up_exps.weight buffer type overriden to CPU
  261. Tensor blk.20.attn_norm.weight buffer type overriden to CUDA0
  262. Tensor blk.20.attn_q.weight buffer type overriden to CUDA0
  263. Tensor blk.20.attn_k.weight buffer type overriden to CUDA0
  264. Tensor blk.20.attn_v.weight buffer type overriden to CUDA0
  265. Tensor blk.20.attn_output.weight buffer type overriden to CUDA0
  266. Tensor blk.20.attn_k_norm.weight buffer type overriden to CUDA0
  267. Tensor blk.20.attn_q_norm.weight buffer type overriden to CUDA0
  268. Tensor blk.20.ffn_norm.weight buffer type overriden to CUDA0
  269. Tensor blk.20.ffn_gate_inp.weight buffer type overriden to CUDA0
  270. Tensor blk.20.ffn_gate_exps.weight buffer type overriden to CPU
  271. Tensor blk.20.ffn_down_exps.weight buffer type overriden to CPU
  272. Tensor blk.20.ffn_up_exps.weight buffer type overriden to CPU
  273. Tensor blk.21.attn_norm.weight buffer type overriden to CUDA0
  274. Tensor blk.21.attn_q.weight buffer type overriden to CUDA0
  275. Tensor blk.21.attn_k.weight buffer type overriden to CUDA0
  276. Tensor blk.21.attn_v.weight buffer type overriden to CUDA0
  277. Tensor blk.21.attn_output.weight buffer type overriden to CUDA0
  278. Tensor blk.21.attn_k_norm.weight buffer type overriden to CUDA0
  279. Tensor blk.21.attn_q_norm.weight buffer type overriden to CUDA0
  280. Tensor blk.21.ffn_norm.weight buffer type overriden to CUDA0
  281. Tensor blk.21.ffn_gate_inp.weight buffer type overriden to CUDA0
  282. Tensor blk.21.ffn_gate_exps.weight buffer type overriden to CPU
  283. Tensor blk.21.ffn_down_exps.weight buffer type overriden to CPU
  284. Tensor blk.21.ffn_up_exps.weight buffer type overriden to CPU
  285. Tensor blk.22.attn_norm.weight buffer type overriden to CUDA0
  286. Tensor blk.22.attn_q.weight buffer type overriden to CUDA0
  287. Tensor blk.22.attn_k.weight buffer type overriden to CUDA0
  288. Tensor blk.22.attn_v.weight buffer type overriden to CUDA0
  289. Tensor blk.22.attn_output.weight buffer type overriden to CUDA0
  290. Tensor blk.22.attn_k_norm.weight buffer type overriden to CUDA0
  291. Tensor blk.22.attn_q_norm.weight buffer type overriden to CUDA0
  292. Tensor blk.22.ffn_norm.weight buffer type overriden to CUDA0
  293. Tensor blk.22.ffn_gate_inp.weight buffer type overriden to CUDA0
  294. Tensor blk.22.ffn_gate_exps.weight buffer type overriden to CPU
  295. Tensor blk.22.ffn_down_exps.weight buffer type overriden to CPU
  296. Tensor blk.22.ffn_up_exps.weight buffer type overriden to CPU
  297. Tensor blk.23.attn_norm.weight buffer type overriden to CUDA0
  298. Tensor blk.23.attn_q.weight buffer type overriden to CUDA0
  299. Tensor blk.23.attn_k.weight buffer type overriden to CUDA0
  300. Tensor blk.23.attn_v.weight buffer type overriden to CUDA0
  301. Tensor blk.23.attn_output.weight buffer type overriden to CUDA0
  302. Tensor blk.23.attn_k_norm.weight buffer type overriden to CUDA0
  303. Tensor blk.23.attn_q_norm.weight buffer type overriden to CUDA0
  304. Tensor blk.23.ffn_norm.weight buffer type overriden to CUDA0
  305. Tensor blk.23.ffn_gate_inp.weight buffer type overriden to CUDA0
  306. Tensor blk.23.ffn_gate_exps.weight buffer type overriden to CPU
  307. Tensor blk.23.ffn_down_exps.weight buffer type overriden to CPU
  308. Tensor blk.23.ffn_up_exps.weight buffer type overriden to CPU
  309. Tensor blk.24.attn_norm.weight buffer type overriden to CUDA0
  310. Tensor blk.24.attn_q.weight buffer type overriden to CUDA0
  311. Tensor blk.24.attn_k.weight buffer type overriden to CUDA0
  312. Tensor blk.24.attn_v.weight buffer type overriden to CUDA0
  313. Tensor blk.24.attn_output.weight buffer type overriden to CUDA0
  314. Tensor blk.24.attn_k_norm.weight buffer type overriden to CUDA0
  315. Tensor blk.24.attn_q_norm.weight buffer type overriden to CUDA0
  316. Tensor blk.24.ffn_norm.weight buffer type overriden to CUDA0
  317. Tensor blk.24.ffn_gate_inp.weight buffer type overriden to CUDA0
  318. Tensor blk.24.ffn_gate_exps.weight buffer type overriden to CPU
  319. Tensor blk.24.ffn_down_exps.weight buffer type overriden to CPU
  320. Tensor blk.24.ffn_up_exps.weight buffer type overriden to CPU
  321. Tensor blk.25.attn_norm.weight buffer type overriden to CUDA0
  322. Tensor blk.25.attn_q.weight buffer type overriden to CUDA0
  323. Tensor blk.25.attn_k.weight buffer type overriden to CUDA0
  324. Tensor blk.25.attn_v.weight buffer type overriden to CUDA0
  325. Tensor blk.25.attn_output.weight buffer type overriden to CUDA0
  326. Tensor blk.25.attn_k_norm.weight buffer type overriden to CUDA0
  327. Tensor blk.25.attn_q_norm.weight buffer type overriden to CUDA0
  328. Tensor blk.25.ffn_norm.weight buffer type overriden to CUDA0
  329. Tensor blk.25.ffn_gate_inp.weight buffer type overriden to CUDA0
  330. Tensor blk.25.ffn_gate_exps.weight buffer type overriden to CPU
  331. Tensor blk.25.ffn_down_exps.weight buffer type overriden to CPU
  332. Tensor blk.25.ffn_up_exps.weight buffer type overriden to CPU
  333. Tensor blk.26.attn_norm.weight buffer type overriden to CUDA0
  334. Tensor blk.26.attn_q.weight buffer type overriden to CUDA0
  335. Tensor blk.26.attn_k.weight buffer type overriden to CUDA0
  336. Tensor blk.26.attn_v.weight buffer type overriden to CUDA0
  337. Tensor blk.26.attn_output.weight buffer type overriden to CUDA0
  338. Tensor blk.26.attn_k_norm.weight buffer type overriden to CUDA0
  339. Tensor blk.26.attn_q_norm.weight buffer type overriden to CUDA0
  340. Tensor blk.26.ffn_norm.weight buffer type overriden to CUDA0
  341. Tensor blk.26.ffn_gate_inp.weight buffer type overriden to CUDA0
  342. Tensor blk.26.ffn_gate_exps.weight buffer type overriden to CPU
  343. Tensor blk.26.ffn_down_exps.weight buffer type overriden to CPU
  344. Tensor blk.26.ffn_up_exps.weight buffer type overriden to CPU
  345. Tensor blk.27.attn_norm.weight buffer type overriden to CUDA0
  346. Tensor blk.27.attn_q.weight buffer type overriden to CUDA0
  347. Tensor blk.27.attn_k.weight buffer type overriden to CUDA0
  348. Tensor blk.27.attn_v.weight buffer type overriden to CUDA0
  349. Tensor blk.27.attn_output.weight buffer type overriden to CUDA0
  350. Tensor blk.27.attn_k_norm.weight buffer type overriden to CUDA0
  351. Tensor blk.27.attn_q_norm.weight buffer type overriden to CUDA0
  352. Tensor blk.27.ffn_norm.weight buffer type overriden to CUDA0
  353. Tensor blk.27.ffn_gate_inp.weight buffer type overriden to CUDA0
  354. Tensor blk.27.ffn_gate_exps.weight buffer type overriden to CPU
  355. Tensor blk.27.ffn_down_exps.weight buffer type overriden to CPU
  356. Tensor blk.27.ffn_up_exps.weight buffer type overriden to CPU
  357. Tensor blk.28.attn_norm.weight buffer type overriden to CUDA0
  358. Tensor blk.28.attn_q.weight buffer type overriden to CUDA0
  359. Tensor blk.28.attn_k.weight buffer type overriden to CUDA0
  360. Tensor blk.28.attn_v.weight buffer type overriden to CUDA0
  361. Tensor blk.28.attn_output.weight buffer type overriden to CUDA0
  362. Tensor blk.28.attn_k_norm.weight buffer type overriden to CUDA0
  363. Tensor blk.28.attn_q_norm.weight buffer type overriden to CUDA0
  364. Tensor blk.28.ffn_norm.weight buffer type overriden to CUDA0
  365. Tensor blk.28.ffn_gate_inp.weight buffer type overriden to CUDA0
  366. Tensor blk.28.ffn_gate_exps.weight buffer type overriden to CPU
  367. Tensor blk.28.ffn_down_exps.weight buffer type overriden to CPU
  368. Tensor blk.28.ffn_up_exps.weight buffer type overriden to CPU
  369. Tensor blk.29.attn_norm.weight buffer type overriden to CUDA0
  370. Tensor blk.29.attn_q.weight buffer type overriden to CUDA0
  371. Tensor blk.29.attn_k.weight buffer type overriden to CUDA0
  372. Tensor blk.29.attn_v.weight buffer type overriden to CUDA0
  373. Tensor blk.29.attn_output.weight buffer type overriden to CUDA0
  374. Tensor blk.29.attn_k_norm.weight buffer type overriden to CUDA0
  375. Tensor blk.29.attn_q_norm.weight buffer type overriden to CUDA0
  376. Tensor blk.29.ffn_norm.weight buffer type overriden to CUDA0
  377. Tensor blk.29.ffn_gate_inp.weight buffer type overriden to CUDA0
  378. Tensor blk.29.ffn_gate_exps.weight buffer type overriden to CPU
  379. Tensor blk.29.ffn_down_exps.weight buffer type overriden to CPU
  380. Tensor blk.29.ffn_up_exps.weight buffer type overriden to CPU
  381. Tensor blk.30.attn_norm.weight buffer type overriden to CUDA0
  382. Tensor blk.30.attn_q.weight buffer type overriden to CUDA0
  383. Tensor blk.30.attn_k.weight buffer type overriden to CUDA0
  384. Tensor blk.30.attn_v.weight buffer type overriden to CUDA0
  385. Tensor blk.30.attn_output.weight buffer type overriden to CUDA0
  386. Tensor blk.30.attn_k_norm.weight buffer type overriden to CUDA0
  387. Tensor blk.30.attn_q_norm.weight buffer type overriden to CUDA0
  388. Tensor blk.30.ffn_norm.weight buffer type overriden to CUDA0
  389. Tensor blk.30.ffn_gate_inp.weight buffer type overriden to CUDA0
  390. Tensor blk.30.ffn_gate_exps.weight buffer type overriden to CPU
  391. Tensor blk.30.ffn_down_exps.weight buffer type overriden to CPU
  392. Tensor blk.30.ffn_up_exps.weight buffer type overriden to CPU
  393. Tensor blk.31.attn_norm.weight buffer type overriden to CUDA0
  394. Tensor blk.31.attn_q.weight buffer type overriden to CUDA0
  395. Tensor blk.31.attn_k.weight buffer type overriden to CUDA0
  396. Tensor blk.31.attn_v.weight buffer type overriden to CUDA0
  397. Tensor blk.31.attn_output.weight buffer type overriden to CUDA0
  398. Tensor blk.31.attn_k_norm.weight buffer type overriden to CUDA0
  399. Tensor blk.31.attn_q_norm.weight buffer type overriden to CUDA0
  400. Tensor blk.31.ffn_norm.weight buffer type overriden to CUDA0
  401. Tensor blk.31.ffn_gate_inp.weight buffer type overriden to CUDA0
  402. Tensor blk.31.ffn_gate_exps.weight buffer type overriden to CPU
  403. Tensor blk.31.ffn_down_exps.weight buffer type overriden to CPU
  404. Tensor blk.31.ffn_up_exps.weight buffer type overriden to CPU
  405. Tensor blk.32.attn_norm.weight buffer type overriden to CUDA0
  406. Tensor blk.32.attn_q.weight buffer type overriden to CUDA0
  407. Tensor blk.32.attn_k.weight buffer type overriden to CUDA0
  408. Tensor blk.32.attn_v.weight buffer type overriden to CUDA0
  409. Tensor blk.32.attn_output.weight buffer type overriden to CUDA0
  410. Tensor blk.32.attn_k_norm.weight buffer type overriden to CUDA0
  411. Tensor blk.32.attn_q_norm.weight buffer type overriden to CUDA0
  412. Tensor blk.32.ffn_norm.weight buffer type overriden to CUDA0
  413. Tensor blk.32.ffn_gate_inp.weight buffer type overriden to CUDA0
  414. Tensor blk.32.ffn_gate_exps.weight buffer type overriden to CPU
  415. Tensor blk.32.ffn_down_exps.weight buffer type overriden to CPU
  416. Tensor blk.32.ffn_up_exps.weight buffer type overriden to CPU
  417. Tensor blk.33.attn_norm.weight buffer type overriden to CUDA0
  418. Tensor blk.33.attn_q.weight buffer type overriden to CUDA0
  419. Tensor blk.33.attn_k.weight buffer type overriden to CUDA0
  420. Tensor blk.33.attn_v.weight buffer type overriden to CUDA0
  421. Tensor blk.33.attn_output.weight buffer type overriden to CUDA0
  422. Tensor blk.33.attn_k_norm.weight buffer type overriden to CUDA0
  423. Tensor blk.33.attn_q_norm.weight buffer type overriden to CUDA0
  424. Tensor blk.33.ffn_norm.weight buffer type overriden to CUDA0
  425. Tensor blk.33.ffn_gate_inp.weight buffer type overriden to CUDA0
  426. Tensor blk.33.ffn_gate_exps.weight buffer type overriden to CPU
  427. Tensor blk.33.ffn_down_exps.weight buffer type overriden to CPU
  428. Tensor blk.33.ffn_up_exps.weight buffer type overriden to CPU
  429. Tensor blk.34.attn_norm.weight buffer type overriden to CUDA0
  430. Tensor blk.34.attn_q.weight buffer type overriden to CUDA0
  431. Tensor blk.34.attn_k.weight buffer type overriden to CUDA0
  432. Tensor blk.34.attn_v.weight buffer type overriden to CUDA0
  433. Tensor blk.34.attn_output.weight buffer type overriden to CUDA0
  434. Tensor blk.34.attn_k_norm.weight buffer type overriden to CUDA0
  435. Tensor blk.34.attn_q_norm.weight buffer type overriden to CUDA0
  436. Tensor blk.34.ffn_norm.weight buffer type overriden to CUDA0
  437. Tensor blk.34.ffn_gate_inp.weight buffer type overriden to CUDA0
  438. Tensor blk.34.ffn_gate_exps.weight buffer type overriden to CPU
  439. Tensor blk.34.ffn_down_exps.weight buffer type overriden to CPU
  440. Tensor blk.34.ffn_up_exps.weight buffer type overriden to CPU
  441. Tensor blk.35.attn_norm.weight buffer type overriden to CUDA0
  442. Tensor blk.35.attn_q.weight buffer type overriden to CUDA0
  443. Tensor blk.35.attn_k.weight buffer type overriden to CUDA0
  444. Tensor blk.35.attn_v.weight buffer type overriden to CUDA0
  445. Tensor blk.35.attn_output.weight buffer type overriden to CUDA0
  446. Tensor blk.35.attn_k_norm.weight buffer type overriden to CUDA0
  447. Tensor blk.35.attn_q_norm.weight buffer type overriden to CUDA0
  448. Tensor blk.35.ffn_norm.weight buffer type overriden to CUDA0
  449. Tensor blk.35.ffn_gate_inp.weight buffer type overriden to CUDA0
  450. Tensor blk.35.ffn_gate_exps.weight buffer type overriden to CPU
  451. Tensor blk.35.ffn_down_exps.weight buffer type overriden to CPU
  452. Tensor blk.35.ffn_up_exps.weight buffer type overriden to CPU
  453. Tensor blk.36.attn_norm.weight buffer type overriden to CUDA0
  454. Tensor blk.36.attn_q.weight buffer type overriden to CUDA0
  455. Tensor blk.36.attn_k.weight buffer type overriden to CUDA0
  456. Tensor blk.36.attn_v.weight buffer type overriden to CUDA0
  457. Tensor blk.36.attn_output.weight buffer type overriden to CUDA0
  458. Tensor blk.36.attn_k_norm.weight buffer type overriden to CUDA0
  459. Tensor blk.36.attn_q_norm.weight buffer type overriden to CUDA0
  460. Tensor blk.36.ffn_norm.weight buffer type overriden to CUDA0
  461. Tensor blk.36.ffn_gate_inp.weight buffer type overriden to CUDA0
  462. Tensor blk.36.ffn_gate_exps.weight buffer type overriden to CPU
  463. Tensor blk.36.ffn_down_exps.weight buffer type overriden to CPU
  464. Tensor blk.36.ffn_up_exps.weight buffer type overriden to CPU
  465. Tensor blk.37.attn_norm.weight buffer type overriden to CUDA0
  466. Tensor blk.37.attn_q.weight buffer type overriden to CUDA0
  467. Tensor blk.37.attn_k.weight buffer type overriden to CUDA0
  468. Tensor blk.37.attn_v.weight buffer type overriden to CUDA0
  469. Tensor blk.37.attn_output.weight buffer type overriden to CUDA0
  470. Tensor blk.37.attn_k_norm.weight buffer type overriden to CUDA0
  471. Tensor blk.37.attn_q_norm.weight buffer type overriden to CUDA0
  472. Tensor blk.37.ffn_norm.weight buffer type overriden to CUDA0
  473. Tensor blk.37.ffn_gate_inp.weight buffer type overriden to CUDA0
  474. Tensor blk.37.ffn_gate_exps.weight buffer type overriden to CPU
  475. Tensor blk.37.ffn_down_exps.weight buffer type overriden to CPU
  476. Tensor blk.37.ffn_up_exps.weight buffer type overriden to CPU
  477. Tensor blk.38.attn_norm.weight buffer type overriden to CUDA0
  478. Tensor blk.38.attn_q.weight buffer type overriden to CUDA0
  479. Tensor blk.38.attn_k.weight buffer type overriden to CUDA0
  480. Tensor blk.38.attn_v.weight buffer type overriden to CUDA0
  481. Tensor blk.38.attn_output.weight buffer type overriden to CUDA0
  482. Tensor blk.38.attn_k_norm.weight buffer type overriden to CUDA0
  483. Tensor blk.38.attn_q_norm.weight buffer type overriden to CUDA0
  484. Tensor blk.38.ffn_norm.weight buffer type overriden to CUDA0
  485. Tensor blk.38.ffn_gate_inp.weight buffer type overriden to CUDA0
  486. Tensor blk.38.ffn_gate_exps.weight buffer type overriden to CPU
  487. Tensor blk.38.ffn_down_exps.weight buffer type overriden to CPU
  488. Tensor blk.38.ffn_up_exps.weight buffer type overriden to CPU
  489. Tensor blk.39.attn_norm.weight buffer type overriden to CUDA0
  490. Tensor blk.39.attn_q.weight buffer type overriden to CUDA0
  491. Tensor blk.39.attn_k.weight buffer type overriden to CUDA0
  492. Tensor blk.39.attn_v.weight buffer type overriden to CUDA0
  493. Tensor blk.39.attn_output.weight buffer type overriden to CUDA0
  494. Tensor blk.39.attn_k_norm.weight buffer type overriden to CUDA0
  495. Tensor blk.39.attn_q_norm.weight buffer type overriden to CUDA0
  496. Tensor blk.39.ffn_norm.weight buffer type overriden to CUDA0
  497. Tensor blk.39.ffn_gate_inp.weight buffer type overriden to CUDA0
  498. Tensor blk.39.ffn_gate_exps.weight buffer type overriden to CPU
  499. Tensor blk.39.ffn_down_exps.weight buffer type overriden to CPU
  500. Tensor blk.39.ffn_up_exps.weight buffer type overriden to CPU
  501. Tensor blk.40.attn_norm.weight buffer type overriden to CUDA0
  502. Tensor blk.40.attn_q.weight buffer type overriden to CUDA0
  503. Tensor blk.40.attn_k.weight buffer type overriden to CUDA0
  504. Tensor blk.40.attn_v.weight buffer type overriden to CUDA0
  505. Tensor blk.40.attn_output.weight buffer type overriden to CUDA0
  506. Tensor blk.40.attn_k_norm.weight buffer type overriden to CUDA0
  507. Tensor blk.40.attn_q_norm.weight buffer type overriden to CUDA0
  508. Tensor blk.40.ffn_norm.weight buffer type overriden to CUDA0
  509. Tensor blk.40.ffn_gate_inp.weight buffer type overriden to CUDA0
  510. Tensor blk.40.ffn_gate_exps.weight buffer type overriden to CPU
  511. Tensor blk.40.ffn_down_exps.weight buffer type overriden to CPU
  512. Tensor blk.40.ffn_up_exps.weight buffer type overriden to CPU
  513. Tensor blk.41.attn_norm.weight buffer type overriden to CUDA0
  514. Tensor blk.41.attn_q.weight buffer type overriden to CUDA0
  515. Tensor blk.41.attn_k.weight buffer type overriden to CUDA0
  516. Tensor blk.41.attn_v.weight buffer type overriden to CUDA0
  517. Tensor blk.41.attn_output.weight buffer type overriden to CUDA0
  518. Tensor blk.41.attn_k_norm.weight buffer type overriden to CUDA0
  519. Tensor blk.41.attn_q_norm.weight buffer type overriden to CUDA0
  520. Tensor blk.41.ffn_norm.weight buffer type overriden to CUDA0
  521. Tensor blk.41.ffn_gate_inp.weight buffer type overriden to CUDA0
  522. Tensor blk.41.ffn_gate_exps.weight buffer type overriden to CPU
  523. Tensor blk.41.ffn_down_exps.weight buffer type overriden to CPU
  524. Tensor blk.41.ffn_up_exps.weight buffer type overriden to CPU
  525. Tensor blk.42.attn_norm.weight buffer type overriden to CUDA0
  526. Tensor blk.42.attn_q.weight buffer type overriden to CUDA0
  527. Tensor blk.42.attn_k.weight buffer type overriden to CUDA0
  528. Tensor blk.42.attn_v.weight buffer type overriden to CUDA0
  529. Tensor blk.42.attn_output.weight buffer type overriden to CUDA0
  530. Tensor blk.42.attn_k_norm.weight buffer type overriden to CUDA0
  531. Tensor blk.42.attn_q_norm.weight buffer type overriden to CUDA0
  532. Tensor blk.42.ffn_norm.weight buffer type overriden to CUDA0
  533. Tensor blk.42.ffn_gate_inp.weight buffer type overriden to CUDA0
  534. Tensor blk.42.ffn_gate_exps.weight buffer type overriden to CPU
  535. Tensor blk.42.ffn_down_exps.weight buffer type overriden to CPU
  536. Tensor blk.42.ffn_up_exps.weight buffer type overriden to CPU
  537. Tensor blk.43.attn_norm.weight buffer type overriden to CUDA0
  538. Tensor blk.43.attn_q.weight buffer type overriden to CUDA0
  539. Tensor blk.43.attn_k.weight buffer type overriden to CUDA0
  540. Tensor blk.43.attn_v.weight buffer type overriden to CUDA0
  541. Tensor blk.43.attn_output.weight buffer type overriden to CUDA0
  542. Tensor blk.43.attn_k_norm.weight buffer type overriden to CUDA0
  543. Tensor blk.43.attn_q_norm.weight buffer type overriden to CUDA0
  544. Tensor blk.43.ffn_norm.weight buffer type overriden to CUDA0
  545. Tensor blk.43.ffn_gate_inp.weight buffer type overriden to CUDA0
  546. Tensor blk.43.ffn_gate_exps.weight buffer type overriden to CPU
  547. Tensor blk.43.ffn_down_exps.weight buffer type overriden to CPU
  548. Tensor blk.43.ffn_up_exps.weight buffer type overriden to CPU
  549. Tensor blk.44.attn_norm.weight buffer type overriden to CUDA0
  550. Tensor blk.44.attn_q.weight buffer type overriden to CUDA0
  551. Tensor blk.44.attn_k.weight buffer type overriden to CUDA0
  552. Tensor blk.44.attn_v.weight buffer type overriden to CUDA0
  553. Tensor blk.44.attn_output.weight buffer type overriden to CUDA0
  554. Tensor blk.44.attn_k_norm.weight buffer type overriden to CUDA0
  555. Tensor blk.44.attn_q_norm.weight buffer type overriden to CUDA0
  556. Tensor blk.44.ffn_norm.weight buffer type overriden to CUDA0
  557. Tensor blk.44.ffn_gate_inp.weight buffer type overriden to CUDA0
  558. Tensor blk.44.ffn_gate_exps.weight buffer type overriden to CPU
  559. Tensor blk.44.ffn_down_exps.weight buffer type overriden to CPU
  560. Tensor blk.44.ffn_up_exps.weight buffer type overriden to CPU
  561. Tensor blk.45.attn_norm.weight buffer type overriden to CUDA0
  562. Tensor blk.45.attn_q.weight buffer type overriden to CUDA0
  563. Tensor blk.45.attn_k.weight buffer type overriden to CUDA0
  564. Tensor blk.45.attn_v.weight buffer type overriden to CUDA0
  565. Tensor blk.45.attn_output.weight buffer type overriden to CUDA0
  566. Tensor blk.45.attn_k_norm.weight buffer type overriden to CUDA0
  567. Tensor blk.45.attn_q_norm.weight buffer type overriden to CUDA0
  568. Tensor blk.45.ffn_norm.weight buffer type overriden to CUDA0
  569. Tensor blk.45.ffn_gate_inp.weight buffer type overriden to CUDA0
  570. Tensor blk.45.ffn_gate_exps.weight buffer type overriden to CPU
  571. Tensor blk.45.ffn_down_exps.weight buffer type overriden to CPU
  572. Tensor blk.45.ffn_up_exps.weight buffer type overriden to CPU
  573. Tensor blk.46.attn_norm.weight buffer type overriden to CUDA0
  574. Tensor blk.46.attn_q.weight buffer type overriden to CUDA0
  575. Tensor blk.46.attn_k.weight buffer type overriden to CUDA0
  576. Tensor blk.46.attn_v.weight buffer type overriden to CUDA0
  577. Tensor blk.46.attn_output.weight buffer type overriden to CUDA0
  578. Tensor blk.46.attn_k_norm.weight buffer type overriden to CUDA0
  579. Tensor blk.46.attn_q_norm.weight buffer type overriden to CUDA0
  580. Tensor blk.46.ffn_norm.weight buffer type overriden to CUDA0
  581. Tensor blk.46.ffn_gate_inp.weight buffer type overriden to CUDA0
  582. Tensor blk.46.ffn_gate_exps.weight buffer type overriden to CPU
  583. Tensor blk.46.ffn_down_exps.weight buffer type overriden to CPU
  584. Tensor blk.46.ffn_up_exps.weight buffer type overriden to CPU
  585. Tensor blk.47.attn_norm.weight buffer type overriden to CUDA1
  586. Tensor blk.47.attn_q.weight buffer type overriden to CUDA1
  587. Tensor blk.47.attn_k.weight buffer type overriden to CUDA1
  588. Tensor blk.47.attn_v.weight buffer type overriden to CUDA1
  589. Tensor blk.47.attn_output.weight buffer type overriden to CUDA1
  590. Tensor blk.47.attn_k_norm.weight buffer type overriden to CUDA1
  591. Tensor blk.47.attn_q_norm.weight buffer type overriden to CUDA1
  592. Tensor blk.47.ffn_norm.weight buffer type overriden to CUDA1
  593. Tensor blk.47.ffn_gate_inp.weight buffer type overriden to CUDA1
  594. Tensor blk.47.ffn_gate_exps.weight buffer type overriden to CPU
  595. Tensor blk.47.ffn_down_exps.weight buffer type overriden to CPU
  596. Tensor blk.47.ffn_up_exps.weight buffer type overriden to CPU
  597. Tensor blk.48.attn_norm.weight buffer type overriden to CUDA1
  598. Tensor blk.48.attn_q.weight buffer type overriden to CUDA1
  599. Tensor blk.48.attn_k.weight buffer type overriden to CUDA1
  600. Tensor blk.48.attn_v.weight buffer type overriden to CUDA1
  601. Tensor blk.48.attn_output.weight buffer type overriden to CUDA1
  602. Tensor blk.48.attn_k_norm.weight buffer type overriden to CUDA1
  603. Tensor blk.48.attn_q_norm.weight buffer type overriden to CUDA1
  604. Tensor blk.48.ffn_norm.weight buffer type overriden to CUDA1
  605. Tensor blk.48.ffn_gate_inp.weight buffer type overriden to CUDA1
  606. Tensor blk.48.ffn_gate_exps.weight buffer type overriden to CPU
  607. Tensor blk.48.ffn_down_exps.weight buffer type overriden to CPU
  608. Tensor blk.48.ffn_up_exps.weight buffer type overriden to CPU
  609. Tensor blk.49.attn_norm.weight buffer type overriden to CUDA1
  610. Tensor blk.49.attn_q.weight buffer type overriden to CUDA1
  611. Tensor blk.49.attn_k.weight buffer type overriden to CUDA1
  612. Tensor blk.49.attn_v.weight buffer type overriden to CUDA1
  613. Tensor blk.49.attn_output.weight buffer type overriden to CUDA1
  614. Tensor blk.49.attn_k_norm.weight buffer type overriden to CUDA1
  615. Tensor blk.49.attn_q_norm.weight buffer type overriden to CUDA1
  616. Tensor blk.49.ffn_norm.weight buffer type overriden to CUDA1
  617. Tensor blk.49.ffn_gate_inp.weight buffer type overriden to CUDA1
  618. Tensor blk.49.ffn_gate_exps.weight buffer type overriden to CPU
  619. Tensor blk.49.ffn_down_exps.weight buffer type overriden to CPU
  620. Tensor blk.49.ffn_up_exps.weight buffer type overriden to CPU
  621. Tensor blk.50.attn_norm.weight buffer type overriden to CUDA1
  622. Tensor blk.50.attn_q.weight buffer type overriden to CUDA1
  623. Tensor blk.50.attn_k.weight buffer type overriden to CUDA1
  624. Tensor blk.50.attn_v.weight buffer type overriden to CUDA1
  625. Tensor blk.50.attn_output.weight buffer type overriden to CUDA1
  626. Tensor blk.50.attn_k_norm.weight buffer type overriden to CUDA1
  627. Tensor blk.50.attn_q_norm.weight buffer type overriden to CUDA1
  628. Tensor blk.50.ffn_norm.weight buffer type overriden to CUDA1
  629. Tensor blk.50.ffn_gate_inp.weight buffer type overriden to CUDA1
  630. Tensor blk.50.ffn_gate_exps.weight buffer type overriden to CUDA1
  631. Tensor blk.50.ffn_down_exps.weight buffer type overriden to CUDA1
  632. Tensor blk.50.ffn_up_exps.weight buffer type overriden to CUDA1
  633. Tensor blk.51.attn_norm.weight buffer type overriden to CUDA1
  634. Tensor blk.51.attn_q.weight buffer type overriden to CUDA1
  635. Tensor blk.51.attn_k.weight buffer type overriden to CUDA1
  636. Tensor blk.51.attn_v.weight buffer type overriden to CUDA1
  637. Tensor blk.51.attn_output.weight buffer type overriden to CUDA1
  638. Tensor blk.51.attn_k_norm.weight buffer type overriden to CUDA1
  639. Tensor blk.51.attn_q_norm.weight buffer type overriden to CUDA1
  640. Tensor blk.51.ffn_norm.weight buffer type overriden to CUDA1
  641. Tensor blk.51.ffn_gate_inp.weight buffer type overriden to CUDA1
  642. Tensor blk.51.ffn_gate_exps.weight buffer type overriden to CUDA1
  643. Tensor blk.51.ffn_down_exps.weight buffer type overriden to CUDA1
  644. Tensor blk.51.ffn_up_exps.weight buffer type overriden to CUDA1
  645. Tensor blk.52.attn_norm.weight buffer type overriden to CUDA1
  646. Tensor blk.52.attn_q.weight buffer type overriden to CUDA1
  647. Tensor blk.52.attn_k.weight buffer type overriden to CUDA1
  648. Tensor blk.52.attn_v.weight buffer type overriden to CUDA1
  649. Tensor blk.52.attn_output.weight buffer type overriden to CUDA1
  650. Tensor blk.52.attn_k_norm.weight buffer type overriden to CUDA1
  651. Tensor blk.52.attn_q_norm.weight buffer type overriden to CUDA1
  652. Tensor blk.52.ffn_norm.weight buffer type overriden to CUDA1
  653. Tensor blk.52.ffn_gate_inp.weight buffer type overriden to CUDA1
  654. Tensor blk.52.ffn_gate_exps.weight buffer type overriden to CUDA1
  655. Tensor blk.52.ffn_down_exps.weight buffer type overriden to CUDA1
  656. Tensor blk.52.ffn_up_exps.weight buffer type overriden to CUDA1
  657. Tensor blk.53.attn_norm.weight buffer type overriden to CUDA1
  658. Tensor blk.53.attn_q.weight buffer type overriden to CUDA1
  659. Tensor blk.53.attn_k.weight buffer type overriden to CUDA1
  660. Tensor blk.53.attn_v.weight buffer type overriden to CUDA1
  661. Tensor blk.53.attn_output.weight buffer type overriden to CUDA1
  662. Tensor blk.53.attn_k_norm.weight buffer type overriden to CUDA1
  663. Tensor blk.53.attn_q_norm.weight buffer type overriden to CUDA1
  664. Tensor blk.53.ffn_norm.weight buffer type overriden to CUDA1
  665. Tensor blk.53.ffn_gate_inp.weight buffer type overriden to CUDA1
  666. Tensor blk.53.ffn_gate_exps.weight buffer type overriden to CUDA1
  667. Tensor blk.53.ffn_down_exps.weight buffer type overriden to CUDA1
  668. Tensor blk.53.ffn_up_exps.weight buffer type overriden to CUDA1
  669. Tensor blk.54.attn_norm.weight buffer type overriden to CUDA1
  670. Tensor blk.54.attn_q.weight buffer type overriden to CUDA1
  671. Tensor blk.54.attn_k.weight buffer type overriden to CUDA1
  672. Tensor blk.54.attn_v.weight buffer type overriden to CUDA1
  673. Tensor blk.54.attn_output.weight buffer type overriden to CUDA1
  674. Tensor blk.54.attn_k_norm.weight buffer type overriden to CUDA1
  675. Tensor blk.54.attn_q_norm.weight buffer type overriden to CUDA1
  676. Tensor blk.54.ffn_norm.weight buffer type overriden to CUDA1
  677. Tensor blk.54.ffn_gate_inp.weight buffer type overriden to CUDA1
  678. Tensor blk.54.ffn_gate_exps.weight buffer type overriden to CUDA1
  679. Tensor blk.54.ffn_down_exps.weight buffer type overriden to CUDA1
  680. Tensor blk.54.ffn_up_exps.weight buffer type overriden to CUDA1
  681. Tensor blk.55.attn_norm.weight buffer type overriden to CUDA1
  682. Tensor blk.55.attn_q.weight buffer type overriden to CUDA1
  683. Tensor blk.55.attn_k.weight buffer type overriden to CUDA1
  684. Tensor blk.55.attn_v.weight buffer type overriden to CUDA1
  685. Tensor blk.55.attn_output.weight buffer type overriden to CUDA1
  686. Tensor blk.55.attn_k_norm.weight buffer type overriden to CUDA1
  687. Tensor blk.55.attn_q_norm.weight buffer type overriden to CUDA1
  688. Tensor blk.55.ffn_norm.weight buffer type overriden to CUDA1
  689. Tensor blk.55.ffn_gate_inp.weight buffer type overriden to CUDA1
  690. Tensor blk.55.ffn_gate_exps.weight buffer type overriden to CUDA1
  691. Tensor blk.55.ffn_down_exps.weight buffer type overriden to CUDA1
  692. Tensor blk.55.ffn_up_exps.weight buffer type overriden to CUDA1
  693. Tensor blk.56.attn_norm.weight buffer type overriden to CUDA1
  694. Tensor blk.56.attn_q.weight buffer type overriden to CUDA1
  695. Tensor blk.56.attn_k.weight buffer type overriden to CUDA1
  696. Tensor blk.56.attn_v.weight buffer type overriden to CUDA1
  697. Tensor blk.56.attn_output.weight buffer type overriden to CUDA1
  698. Tensor blk.56.attn_k_norm.weight buffer type overriden to CUDA1
  699. Tensor blk.56.attn_q_norm.weight buffer type overriden to CUDA1
  700. Tensor blk.56.ffn_norm.weight buffer type overriden to CUDA1
  701. Tensor blk.56.ffn_gate_inp.weight buffer type overriden to CUDA1
  702. Tensor blk.56.ffn_gate_exps.weight buffer type overriden to CUDA1
  703. Tensor blk.56.ffn_down_exps.weight buffer type overriden to CUDA1
  704. Tensor blk.56.ffn_up_exps.weight buffer type overriden to CUDA1
  705. Tensor blk.57.attn_norm.weight buffer type overriden to CUDA1
  706. Tensor blk.57.attn_q.weight buffer type overriden to CUDA1
  707. Tensor blk.57.attn_k.weight buffer type overriden to CUDA1
  708. Tensor blk.57.attn_v.weight buffer type overriden to CUDA1
  709. Tensor blk.57.attn_output.weight buffer type overriden to CUDA1
  710. Tensor blk.57.attn_k_norm.weight buffer type overriden to CUDA1
  711. Tensor blk.57.attn_q_norm.weight buffer type overriden to CUDA1
  712. Tensor blk.57.ffn_norm.weight buffer type overriden to CUDA1
  713. Tensor blk.57.ffn_gate_inp.weight buffer type overriden to CUDA1
  714. Tensor blk.57.ffn_gate_exps.weight buffer type overriden to CUDA1
  715. Tensor blk.57.ffn_down_exps.weight buffer type overriden to CUDA1
  716. Tensor blk.57.ffn_up_exps.weight buffer type overriden to CUDA1
  717. Tensor blk.58.attn_norm.weight buffer type overriden to CUDA1
  718. Tensor blk.58.attn_q.weight buffer type overriden to CUDA1
  719. Tensor blk.58.attn_k.weight buffer type overriden to CUDA1
  720. Tensor blk.58.attn_v.weight buffer type overriden to CUDA1
  721. Tensor blk.58.attn_output.weight buffer type overriden to CUDA1
  722. Tensor blk.58.attn_k_norm.weight buffer type overriden to CUDA1
  723. Tensor blk.58.attn_q_norm.weight buffer type overriden to CUDA1
  724. Tensor blk.58.ffn_norm.weight buffer type overriden to CUDA1
  725. Tensor blk.58.ffn_gate_inp.weight buffer type overriden to CUDA1
  726. Tensor blk.58.ffn_gate_exps.weight buffer type overriden to CUDA1
  727. Tensor blk.58.ffn_down_exps.weight buffer type overriden to CUDA1
  728. Tensor blk.58.ffn_up_exps.weight buffer type overriden to CUDA1
  729. Tensor blk.59.attn_norm.weight buffer type overriden to CUDA1
  730. Tensor blk.59.attn_q.weight buffer type overriden to CUDA1
  731. Tensor blk.59.attn_k.weight buffer type overriden to CUDA1
  732. Tensor blk.59.attn_v.weight buffer type overriden to CUDA1
  733. Tensor blk.59.attn_output.weight buffer type overriden to CUDA1
  734. Tensor blk.59.attn_k_norm.weight buffer type overriden to CUDA1
  735. Tensor blk.59.attn_q_norm.weight buffer type overriden to CUDA1
  736. Tensor blk.59.ffn_norm.weight buffer type overriden to CUDA1
  737. Tensor blk.59.ffn_gate_inp.weight buffer type overriden to CUDA1
  738. Tensor blk.59.ffn_gate_exps.weight buffer type overriden to CUDA1
  739. Tensor blk.59.ffn_down_exps.weight buffer type overriden to CUDA1
  740. Tensor blk.59.ffn_up_exps.weight buffer type overriden to CUDA1
  741. Tensor blk.60.attn_norm.weight buffer type overriden to CUDA1
  742. Tensor blk.60.attn_q.weight buffer type overriden to CUDA1
  743. Tensor blk.60.attn_k.weight buffer type overriden to CUDA1
  744. Tensor blk.60.attn_v.weight buffer type overriden to CUDA1
  745. Tensor blk.60.attn_output.weight buffer type overriden to CUDA1
  746. Tensor blk.60.attn_k_norm.weight buffer type overriden to CUDA1
  747. Tensor blk.60.attn_q_norm.weight buffer type overriden to CUDA1
  748. Tensor blk.60.ffn_norm.weight buffer type overriden to CUDA1
  749. Tensor blk.60.ffn_gate_inp.weight buffer type overriden to CUDA1
  750. Tensor blk.60.ffn_gate_exps.weight buffer type overriden to CPU
  751. Tensor blk.60.ffn_down_exps.weight buffer type overriden to CPU
  752. Tensor blk.60.ffn_up_exps.weight buffer type overriden to CPU
  753. Tensor blk.61.attn_norm.weight buffer type overriden to CUDA1
  754. Tensor blk.61.attn_q.weight buffer type overriden to CUDA1
  755. Tensor blk.61.attn_k.weight buffer type overriden to CUDA1
  756. Tensor blk.61.attn_v.weight buffer type overriden to CUDA1
  757. Tensor blk.61.attn_output.weight buffer type overriden to CUDA1
  758. Tensor blk.61.attn_k_norm.weight buffer type overriden to CUDA1
  759. Tensor blk.61.attn_q_norm.weight buffer type overriden to CUDA1
  760. Tensor blk.61.ffn_norm.weight buffer type overriden to CUDA1
  761. Tensor blk.61.ffn_gate_inp.weight buffer type overriden to CUDA1
  762. Tensor blk.61.ffn_gate_exps.weight buffer type overriden to CPU
  763. Tensor blk.61.ffn_down_exps.weight buffer type overriden to CPU
  764. Tensor blk.61.ffn_up_exps.weight buffer type overriden to CPU
  765. Tensor blk.62.attn_norm.weight buffer type overriden to CUDA1
  766. Tensor blk.62.attn_q.weight buffer type overriden to CUDA1
  767. Tensor blk.62.attn_k.weight buffer type overriden to CUDA1
  768. Tensor blk.62.attn_v.weight buffer type overriden to CUDA1
  769. Tensor blk.62.attn_output.weight buffer type overriden to CUDA1
  770. Tensor blk.62.attn_k_norm.weight buffer type overriden to CUDA1
  771. Tensor blk.62.attn_q_norm.weight buffer type overriden to CUDA1
  772. Tensor blk.62.ffn_norm.weight buffer type overriden to CUDA1
  773. Tensor blk.62.ffn_gate_inp.weight buffer type overriden to CUDA1
  774. Tensor blk.62.ffn_gate_exps.weight buffer type overriden to CPU
  775. Tensor blk.62.ffn_down_exps.weight buffer type overriden to CPU
  776. Tensor blk.62.ffn_up_exps.weight buffer type overriden to CPU
  777. Tensor blk.63.attn_norm.weight buffer type overriden to CUDA1
  778. Tensor blk.63.attn_q.weight buffer type overriden to CUDA1
  779. Tensor blk.63.attn_k.weight buffer type overriden to CUDA1
  780. Tensor blk.63.attn_v.weight buffer type overriden to CUDA1
  781. Tensor blk.63.attn_output.weight buffer type overriden to CUDA1
  782. Tensor blk.63.attn_k_norm.weight buffer type overriden to CUDA1
  783. Tensor blk.63.attn_q_norm.weight buffer type overriden to CUDA1
  784. Tensor blk.63.ffn_norm.weight buffer type overriden to CUDA1
  785. Tensor blk.63.ffn_gate_inp.weight buffer type overriden to CUDA1
  786. Tensor blk.63.ffn_gate_exps.weight buffer type overriden to CPU
  787. Tensor blk.63.ffn_down_exps.weight buffer type overriden to CPU
  788. Tensor blk.63.ffn_up_exps.weight buffer type overriden to CPU
  789. Tensor blk.64.attn_norm.weight buffer type overriden to CUDA1
  790. Tensor blk.64.attn_q.weight buffer type overriden to CUDA1
  791. Tensor blk.64.attn_k.weight buffer type overriden to CUDA1
  792. Tensor blk.64.attn_v.weight buffer type overriden to CUDA1
  793. Tensor blk.64.attn_output.weight buffer type overriden to CUDA1
  794. Tensor blk.64.attn_k_norm.weight buffer type overriden to CUDA1
  795. Tensor blk.64.attn_q_norm.weight buffer type overriden to CUDA1
  796. Tensor blk.64.ffn_norm.weight buffer type overriden to CUDA1
  797. Tensor blk.64.ffn_gate_inp.weight buffer type overriden to CUDA1
  798. Tensor blk.64.ffn_gate_exps.weight buffer type overriden to CPU
  799. Tensor blk.64.ffn_down_exps.weight buffer type overriden to CPU
  800. Tensor blk.64.ffn_up_exps.weight buffer type overriden to CPU
  801. Tensor blk.65.attn_norm.weight buffer type overriden to CUDA1
  802. Tensor blk.65.attn_q.weight buffer type overriden to CUDA1
  803. Tensor blk.65.attn_k.weight buffer type overriden to CUDA1
  804. Tensor blk.65.attn_v.weight buffer type overriden to CUDA1
  805. Tensor blk.65.attn_output.weight buffer type overriden to CUDA1
  806. Tensor blk.65.attn_k_norm.weight buffer type overriden to CUDA1
  807. Tensor blk.65.attn_q_norm.weight buffer type overriden to CUDA1
  808. Tensor blk.65.ffn_norm.weight buffer type overriden to CUDA1
  809. Tensor blk.65.ffn_gate_inp.weight buffer type overriden to CUDA1
  810. Tensor blk.65.ffn_gate_exps.weight buffer type overriden to CPU
  811. Tensor blk.65.ffn_down_exps.weight buffer type overriden to CPU
  812. Tensor blk.65.ffn_up_exps.weight buffer type overriden to CPU
  813. Tensor blk.66.attn_norm.weight buffer type overriden to CUDA1
  814. Tensor blk.66.attn_q.weight buffer type overriden to CUDA1
  815. Tensor blk.66.attn_k.weight buffer type overriden to CUDA1
  816. Tensor blk.66.attn_v.weight buffer type overriden to CUDA1
  817. Tensor blk.66.attn_output.weight buffer type overriden to CUDA1
  818. Tensor blk.66.attn_k_norm.weight buffer type overriden to CUDA1
  819. Tensor blk.66.attn_q_norm.weight buffer type overriden to CUDA1
  820. Tensor blk.66.ffn_norm.weight buffer type overriden to CUDA1
  821. Tensor blk.66.ffn_gate_inp.weight buffer type overriden to CUDA1
  822. Tensor blk.66.ffn_gate_exps.weight buffer type overriden to CPU
  823. Tensor blk.66.ffn_down_exps.weight buffer type overriden to CPU
  824. Tensor blk.66.ffn_up_exps.weight buffer type overriden to CPU
  825. Tensor blk.67.attn_norm.weight buffer type overriden to CUDA1
  826. Tensor blk.67.attn_q.weight buffer type overriden to CUDA1
  827. Tensor blk.67.attn_k.weight buffer type overriden to CUDA1
  828. Tensor blk.67.attn_v.weight buffer type overriden to CUDA1
  829. Tensor blk.67.attn_output.weight buffer type overriden to CUDA1
  830. Tensor blk.67.attn_k_norm.weight buffer type overriden to CUDA1
  831. Tensor blk.67.attn_q_norm.weight buffer type overriden to CUDA1
  832. Tensor blk.67.ffn_norm.weight buffer type overriden to CUDA1
  833. Tensor blk.67.ffn_gate_inp.weight buffer type overriden to CUDA1
  834. Tensor blk.67.ffn_gate_exps.weight buffer type overriden to CPU
  835. Tensor blk.67.ffn_down_exps.weight buffer type overriden to CPU
  836. Tensor blk.67.ffn_up_exps.weight buffer type overriden to CPU
  837. Tensor blk.68.attn_norm.weight buffer type overriden to CUDA1
  838. Tensor blk.68.attn_q.weight buffer type overriden to CUDA1
  839. Tensor blk.68.attn_k.weight buffer type overriden to CUDA1
  840. Tensor blk.68.attn_v.weight buffer type overriden to CUDA1
  841. Tensor blk.68.attn_output.weight buffer type overriden to CUDA1
  842. Tensor blk.68.attn_k_norm.weight buffer type overriden to CUDA1
  843. Tensor blk.68.attn_q_norm.weight buffer type overriden to CUDA1
  844. Tensor blk.68.ffn_norm.weight buffer type overriden to CUDA1
  845. Tensor blk.68.ffn_gate_inp.weight buffer type overriden to CUDA1
  846. Tensor blk.68.ffn_gate_exps.weight buffer type overriden to CPU
  847. Tensor blk.68.ffn_down_exps.weight buffer type overriden to CPU
  848. Tensor blk.68.ffn_up_exps.weight buffer type overriden to CPU
  849. Tensor blk.69.attn_norm.weight buffer type overriden to CUDA1
  850. Tensor blk.69.attn_q.weight buffer type overriden to CUDA1
  851. Tensor blk.69.attn_k.weight buffer type overriden to CUDA1
  852. Tensor blk.69.attn_v.weight buffer type overriden to CUDA1
  853. Tensor blk.69.attn_output.weight buffer type overriden to CUDA1
  854. Tensor blk.69.attn_k_norm.weight buffer type overriden to CUDA1
  855. Tensor blk.69.attn_q_norm.weight buffer type overriden to CUDA1
  856. Tensor blk.69.ffn_norm.weight buffer type overriden to CUDA1
  857. Tensor blk.69.ffn_gate_inp.weight buffer type overriden to CUDA1
  858. Tensor blk.69.ffn_gate_exps.weight buffer type overriden to CPU
  859. Tensor blk.69.ffn_down_exps.weight buffer type overriden to CPU
  860. Tensor blk.69.ffn_up_exps.weight buffer type overriden to CPU
  861. Tensor blk.70.attn_norm.weight buffer type overriden to CUDA1
  862. Tensor blk.70.attn_q.weight buffer type overriden to CUDA1
  863. Tensor blk.70.attn_k.weight buffer type overriden to CUDA1
  864. Tensor blk.70.attn_v.weight buffer type overriden to CUDA1
  865. Tensor blk.70.attn_output.weight buffer type overriden to CUDA1
  866. Tensor blk.70.attn_k_norm.weight buffer type overriden to CUDA1
  867. Tensor blk.70.attn_q_norm.weight buffer type overriden to CUDA1
  868. Tensor blk.70.ffn_norm.weight buffer type overriden to CUDA1
  869. Tensor blk.70.ffn_gate_inp.weight buffer type overriden to CUDA1
  870. Tensor blk.70.ffn_gate_exps.weight buffer type overriden to CPU
  871. Tensor blk.70.ffn_down_exps.weight buffer type overriden to CPU
  872. Tensor blk.70.ffn_up_exps.weight buffer type overriden to CPU
  873. Tensor blk.71.attn_norm.weight buffer type overriden to CUDA1
  874. Tensor blk.71.attn_q.weight buffer type overriden to CUDA1
  875. Tensor blk.71.attn_k.weight buffer type overriden to CUDA1
  876. Tensor blk.71.attn_v.weight buffer type overriden to CUDA1
  877. Tensor blk.71.attn_output.weight buffer type overriden to CUDA1
  878. Tensor blk.71.attn_k_norm.weight buffer type overriden to CUDA1
  879. Tensor blk.71.attn_q_norm.weight buffer type overriden to CUDA1
  880. Tensor blk.71.ffn_norm.weight buffer type overriden to CUDA1
  881. Tensor blk.71.ffn_gate_inp.weight buffer type overriden to CUDA1
  882. Tensor blk.71.ffn_gate_exps.weight buffer type overriden to CPU
  883. Tensor blk.71.ffn_down_exps.weight buffer type overriden to CPU
  884. Tensor blk.71.ffn_up_exps.weight buffer type overriden to CPU
  885. Tensor blk.72.attn_norm.weight buffer type overriden to CUDA1
  886. Tensor blk.72.attn_q.weight buffer type overriden to CUDA1
  887. Tensor blk.72.attn_k.weight buffer type overriden to CUDA1
  888. Tensor blk.72.attn_v.weight buffer type overriden to CUDA1
  889. Tensor blk.72.attn_output.weight buffer type overriden to CUDA1
  890. Tensor blk.72.attn_k_norm.weight buffer type overriden to CUDA1
  891. Tensor blk.72.attn_q_norm.weight buffer type overriden to CUDA1
  892. Tensor blk.72.ffn_norm.weight buffer type overriden to CUDA1
  893. Tensor blk.72.ffn_gate_inp.weight buffer type overriden to CUDA1
  894. Tensor blk.72.ffn_gate_exps.weight buffer type overriden to CPU
  895. Tensor blk.72.ffn_down_exps.weight buffer type overriden to CPU
  896. Tensor blk.72.ffn_up_exps.weight buffer type overriden to CPU
  897. Tensor blk.73.attn_norm.weight buffer type overriden to CUDA1
  898. Tensor blk.73.attn_q.weight buffer type overriden to CUDA1
  899. Tensor blk.73.attn_k.weight buffer type overriden to CUDA1
  900. Tensor blk.73.attn_v.weight buffer type overriden to CUDA1
  901. Tensor blk.73.attn_output.weight buffer type overriden to CUDA1
  902. Tensor blk.73.attn_k_norm.weight buffer type overriden to CUDA1
  903. Tensor blk.73.attn_q_norm.weight buffer type overriden to CUDA1
  904. Tensor blk.73.ffn_norm.weight buffer type overriden to CUDA1
  905. Tensor blk.73.ffn_gate_inp.weight buffer type overriden to CUDA1
  906. Tensor blk.73.ffn_gate_exps.weight buffer type overriden to CPU
  907. Tensor blk.73.ffn_down_exps.weight buffer type overriden to CPU
  908. Tensor blk.73.ffn_up_exps.weight buffer type overriden to CPU
  909. Tensor blk.74.attn_norm.weight buffer type overriden to CUDA1
  910. Tensor blk.74.attn_q.weight buffer type overriden to CUDA1
  911. Tensor blk.74.attn_k.weight buffer type overriden to CUDA1
  912. Tensor blk.74.attn_v.weight buffer type overriden to CUDA1
  913. Tensor blk.74.attn_output.weight buffer type overriden to CUDA1
  914. Tensor blk.74.attn_k_norm.weight buffer type overriden to CUDA1
  915. Tensor blk.74.attn_q_norm.weight buffer type overriden to CUDA1
  916. Tensor blk.74.ffn_norm.weight buffer type overriden to CUDA1
  917. Tensor blk.74.ffn_gate_inp.weight buffer type overriden to CUDA1
  918. Tensor blk.74.ffn_gate_exps.weight buffer type overriden to CPU
  919. Tensor blk.74.ffn_down_exps.weight buffer type overriden to CPU
  920. Tensor blk.74.ffn_up_exps.weight buffer type overriden to CPU
  921. Tensor blk.75.attn_norm.weight buffer type overriden to CUDA1
  922. Tensor blk.75.attn_q.weight buffer type overriden to CUDA1
  923. Tensor blk.75.attn_k.weight buffer type overriden to CUDA1
  924. Tensor blk.75.attn_v.weight buffer type overriden to CUDA1
  925. Tensor blk.75.attn_output.weight buffer type overriden to CUDA1
  926. Tensor blk.75.attn_k_norm.weight buffer type overriden to CUDA1
  927. Tensor blk.75.attn_q_norm.weight buffer type overriden to CUDA1
  928. Tensor blk.75.ffn_norm.weight buffer type overriden to CUDA1
  929. Tensor blk.75.ffn_gate_inp.weight buffer type overriden to CUDA1
  930. Tensor blk.75.ffn_gate_exps.weight buffer type overriden to CPU
  931. Tensor blk.75.ffn_down_exps.weight buffer type overriden to CPU
  932. Tensor blk.75.ffn_up_exps.weight buffer type overriden to CPU
  933. Tensor blk.76.attn_norm.weight buffer type overriden to CUDA1
  934. Tensor blk.76.attn_q.weight buffer type overriden to CUDA1
  935. Tensor blk.76.attn_k.weight buffer type overriden to CUDA1
  936. Tensor blk.76.attn_v.weight buffer type overriden to CUDA1
  937. Tensor blk.76.attn_output.weight buffer type overriden to CUDA1
  938. Tensor blk.76.attn_k_norm.weight buffer type overriden to CUDA1
  939. Tensor blk.76.attn_q_norm.weight buffer type overriden to CUDA1
  940. Tensor blk.76.ffn_norm.weight buffer type overriden to CUDA1
  941. Tensor blk.76.ffn_gate_inp.weight buffer type overriden to CUDA1
  942. Tensor blk.76.ffn_gate_exps.weight buffer type overriden to CPU
  943. Tensor blk.76.ffn_down_exps.weight buffer type overriden to CPU
  944. Tensor blk.76.ffn_up_exps.weight buffer type overriden to CPU
  945. Tensor blk.77.attn_norm.weight buffer type overriden to CUDA1
  946. Tensor blk.77.attn_q.weight buffer type overriden to CUDA1
  947. Tensor blk.77.attn_k.weight buffer type overriden to CUDA1
  948. Tensor blk.77.attn_v.weight buffer type overriden to CUDA1
  949. Tensor blk.77.attn_output.weight buffer type overriden to CUDA1
  950. Tensor blk.77.attn_k_norm.weight buffer type overriden to CUDA1
  951. Tensor blk.77.attn_q_norm.weight buffer type overriden to CUDA1
  952. Tensor blk.77.ffn_norm.weight buffer type overriden to CUDA1
  953. Tensor blk.77.ffn_gate_inp.weight buffer type overriden to CUDA1
  954. Tensor blk.77.ffn_gate_exps.weight buffer type overriden to CPU
  955. Tensor blk.77.ffn_down_exps.weight buffer type overriden to CPU
  956. Tensor blk.77.ffn_up_exps.weight buffer type overriden to CPU
  957. Tensor blk.78.attn_norm.weight buffer type overriden to CUDA1
  958. Tensor blk.78.attn_q.weight buffer type overriden to CUDA1
  959. Tensor blk.78.attn_k.weight buffer type overriden to CUDA1
  960. Tensor blk.78.attn_v.weight buffer type overriden to CUDA1
  961. Tensor blk.78.attn_output.weight buffer type overriden to CUDA1
  962. Tensor blk.78.attn_k_norm.weight buffer type overriden to CUDA1
  963. Tensor blk.78.attn_q_norm.weight buffer type overriden to CUDA1
  964. Tensor blk.78.ffn_norm.weight buffer type overriden to CUDA1
  965. Tensor blk.78.ffn_gate_inp.weight buffer type overriden to CUDA1
  966. Tensor blk.78.ffn_gate_exps.weight buffer type overriden to CPU
  967. Tensor blk.78.ffn_down_exps.weight buffer type overriden to CPU
  968. Tensor blk.78.ffn_up_exps.weight buffer type overriden to CPU
  969. Tensor blk.79.attn_norm.weight buffer type overriden to CUDA1
  970. Tensor blk.79.attn_q.weight buffer type overriden to CUDA1
  971. Tensor blk.79.attn_k.weight buffer type overriden to CUDA1
  972. Tensor blk.79.attn_v.weight buffer type overriden to CUDA1
  973. Tensor blk.79.attn_output.weight buffer type overriden to CUDA1
  974. Tensor blk.79.attn_k_norm.weight buffer type overriden to CUDA1
  975. Tensor blk.79.attn_q_norm.weight buffer type overriden to CUDA1
  976. Tensor blk.79.ffn_norm.weight buffer type overriden to CUDA1
  977. Tensor blk.79.ffn_gate_inp.weight buffer type overriden to CUDA1
  978. Tensor blk.79.ffn_gate_exps.weight buffer type overriden to CPU
  979. Tensor blk.79.ffn_down_exps.weight buffer type overriden to CPU
  980. Tensor blk.79.ffn_up_exps.weight buffer type overriden to CPU
  981. Tensor blk.80.attn_norm.weight buffer type overriden to CUDA1
  982. Tensor blk.80.attn_q.weight buffer type overriden to CUDA1
  983. Tensor blk.80.attn_k.weight buffer type overriden to CUDA1
  984. Tensor blk.80.attn_v.weight buffer type overriden to CUDA1
  985. Tensor blk.80.attn_output.weight buffer type overriden to CUDA1
  986. Tensor blk.80.attn_k_norm.weight buffer type overriden to CUDA1
  987. Tensor blk.80.attn_q_norm.weight buffer type overriden to CUDA1
  988. Tensor blk.80.ffn_norm.weight buffer type overriden to CUDA1
  989. Tensor blk.80.ffn_gate_inp.weight buffer type overriden to CUDA1
  990. Tensor blk.80.ffn_gate_exps.weight buffer type overriden to CPU
  991. Tensor blk.80.ffn_down_exps.weight buffer type overriden to CPU
  992. Tensor blk.80.ffn_up_exps.weight buffer type overriden to CPU
  993. Tensor blk.81.attn_norm.weight buffer type overriden to CUDA1
  994. Tensor blk.81.attn_q.weight buffer type overriden to CUDA1
  995. Tensor blk.81.attn_k.weight buffer type overriden to CUDA1
  996. Tensor blk.81.attn_v.weight buffer type overriden to CUDA1
  997. Tensor blk.81.attn_output.weight buffer type overriden to CUDA1
  998. Tensor blk.81.attn_k_norm.weight buffer type overriden to CUDA1
  999. Tensor blk.81.attn_q_norm.weight buffer type overriden to CUDA1
  1000. Tensor blk.81.ffn_norm.weight buffer type overriden to CUDA1
  1001. Tensor blk.81.ffn_gate_inp.weight buffer type overriden to CUDA1
  1002. Tensor blk.81.ffn_gate_exps.weight buffer type overriden to CPU
  1003. Tensor blk.81.ffn_down_exps.weight buffer type overriden to CPU
  1004. Tensor blk.81.ffn_up_exps.weight buffer type overriden to CPU
  1005. Tensor blk.82.attn_norm.weight buffer type overriden to CUDA1
  1006. Tensor blk.82.attn_q.weight buffer type overriden to CUDA1
  1007. Tensor blk.82.attn_k.weight buffer type overriden to CUDA1
  1008. Tensor blk.82.attn_v.weight buffer type overriden to CUDA1
  1009. Tensor blk.82.attn_output.weight buffer type overriden to CUDA1
  1010. Tensor blk.82.attn_k_norm.weight buffer type overriden to CUDA1
  1011. Tensor blk.82.attn_q_norm.weight buffer type overriden to CUDA1
  1012. Tensor blk.82.ffn_norm.weight buffer type overriden to CUDA1
  1013. Tensor blk.82.ffn_gate_inp.weight buffer type overriden to CUDA1
  1014. Tensor blk.82.ffn_gate_exps.weight buffer type overriden to CPU
  1015. Tensor blk.82.ffn_down_exps.weight buffer type overriden to CPU
  1016. Tensor blk.82.ffn_up_exps.weight buffer type overriden to CPU
  1017. Tensor blk.83.attn_norm.weight buffer type overriden to CUDA1
  1018. Tensor blk.83.attn_q.weight buffer type overriden to CUDA1
  1019. Tensor blk.83.attn_k.weight buffer type overriden to CUDA1
  1020. Tensor blk.83.attn_v.weight buffer type overriden to CUDA1
  1021. Tensor blk.83.attn_output.weight buffer type overriden to CUDA1
  1022. Tensor blk.83.attn_k_norm.weight buffer type overriden to CUDA1
  1023. Tensor blk.83.attn_q_norm.weight buffer type overriden to CUDA1
  1024. Tensor blk.83.ffn_norm.weight buffer type overriden to CUDA1
  1025. Tensor blk.83.ffn_gate_inp.weight buffer type overriden to CUDA1
  1026. Tensor blk.83.ffn_gate_exps.weight buffer type overriden to CPU
  1027. Tensor blk.83.ffn_down_exps.weight buffer type overriden to CPU
  1028. Tensor blk.83.ffn_up_exps.weight buffer type overriden to CPU
  1029. Tensor blk.84.attn_norm.weight buffer type overriden to CUDA1
  1030. Tensor blk.84.attn_q.weight buffer type overriden to CUDA1
  1031. Tensor blk.84.attn_k.weight buffer type overriden to CUDA1
  1032. Tensor blk.84.attn_v.weight buffer type overriden to CUDA1
  1033. Tensor blk.84.attn_output.weight buffer type overriden to CUDA1
  1034. Tensor blk.84.attn_k_norm.weight buffer type overriden to CUDA1
  1035. Tensor blk.84.attn_q_norm.weight buffer type overriden to CUDA1
  1036. Tensor blk.84.ffn_norm.weight buffer type overriden to CUDA1
  1037. Tensor blk.84.ffn_gate_inp.weight buffer type overriden to CUDA1
  1038. Tensor blk.84.ffn_gate_exps.weight buffer type overriden to CPU
  1039. Tensor blk.84.ffn_down_exps.weight buffer type overriden to CPU
  1040. Tensor blk.84.ffn_up_exps.weight buffer type overriden to CPU
  1041. Tensor blk.85.attn_norm.weight buffer type overriden to CUDA1
  1042. Tensor blk.85.attn_q.weight buffer type overriden to CUDA1
  1043. Tensor blk.85.attn_k.weight buffer type overriden to CUDA1
  1044. Tensor blk.85.attn_v.weight buffer type overriden to CUDA1
  1045. Tensor blk.85.attn_output.weight buffer type overriden to CUDA1
  1046. Tensor blk.85.attn_k_norm.weight buffer type overriden to CUDA1
  1047. Tensor blk.85.attn_q_norm.weight buffer type overriden to CUDA1
  1048. Tensor blk.85.ffn_norm.weight buffer type overriden to CUDA1
  1049. Tensor blk.85.ffn_gate_inp.weight buffer type overriden to CUDA1
  1050. Tensor blk.85.ffn_gate_exps.weight buffer type overriden to CPU
  1051. Tensor blk.85.ffn_down_exps.weight buffer type overriden to CPU
  1052. Tensor blk.85.ffn_up_exps.weight buffer type overriden to CPU
  1053. Tensor blk.86.attn_norm.weight buffer type overriden to CUDA1
  1054. Tensor blk.86.attn_q.weight buffer type overriden to CUDA1
  1055. Tensor blk.86.attn_k.weight buffer type overriden to CUDA1
  1056. Tensor blk.86.attn_v.weight buffer type overriden to CUDA1
  1057. Tensor blk.86.attn_output.weight buffer type overriden to CUDA1
  1058. Tensor blk.86.attn_k_norm.weight buffer type overriden to CUDA1
  1059. Tensor blk.86.attn_q_norm.weight buffer type overriden to CUDA1
  1060. Tensor blk.86.ffn_norm.weight buffer type overriden to CUDA1
  1061. Tensor blk.86.ffn_gate_inp.weight buffer type overriden to CUDA1
  1062. Tensor blk.86.ffn_gate_exps.weight buffer type overriden to CPU
  1063. Tensor blk.86.ffn_down_exps.weight buffer type overriden to CPU
  1064. Tensor blk.86.ffn_up_exps.weight buffer type overriden to CPU
  1065. Tensor blk.87.attn_norm.weight buffer type overriden to CUDA1
  1066. Tensor blk.87.attn_q.weight buffer type overriden to CUDA1
  1067. Tensor blk.87.attn_k.weight buffer type overriden to CUDA1
  1068. Tensor blk.87.attn_v.weight buffer type overriden to CUDA1
  1069. Tensor blk.87.attn_output.weight buffer type overriden to CUDA1
  1070. Tensor blk.87.attn_k_norm.weight buffer type overriden to CUDA1
  1071. Tensor blk.87.attn_q_norm.weight buffer type overriden to CUDA1
  1072. Tensor blk.87.ffn_norm.weight buffer type overriden to CUDA1
  1073. Tensor blk.87.ffn_gate_inp.weight buffer type overriden to CUDA1
  1074. Tensor blk.87.ffn_gate_exps.weight buffer type overriden to CPU
  1075. Tensor blk.87.ffn_down_exps.weight buffer type overriden to CPU
  1076. Tensor blk.87.ffn_up_exps.weight buffer type overriden to CPU
  1077. Tensor blk.88.attn_norm.weight buffer type overriden to CUDA1
  1078. Tensor blk.88.attn_q.weight buffer type overriden to CUDA1
  1079. Tensor blk.88.attn_k.weight buffer type overriden to CUDA1
  1080. Tensor blk.88.attn_v.weight buffer type overriden to CUDA1
  1081. Tensor blk.88.attn_output.weight buffer type overriden to CUDA1
  1082. Tensor blk.88.attn_k_norm.weight buffer type overriden to CUDA1
  1083. Tensor blk.88.attn_q_norm.weight buffer type overriden to CUDA1
  1084. Tensor blk.88.ffn_norm.weight buffer type overriden to CUDA1
  1085. Tensor blk.88.ffn_gate_inp.weight buffer type overriden to CUDA1
  1086. Tensor blk.88.ffn_gate_exps.weight buffer type overriden to CPU
  1087. Tensor blk.88.ffn_down_exps.weight buffer type overriden to CPU
  1088. Tensor blk.88.ffn_up_exps.weight buffer type overriden to CPU
  1089. Tensor blk.89.attn_norm.weight buffer type overriden to CUDA1
  1090. Tensor blk.89.attn_q.weight buffer type overriden to CUDA1
  1091. Tensor blk.89.attn_k.weight buffer type overriden to CUDA1
  1092. Tensor blk.89.attn_v.weight buffer type overriden to CUDA1
  1093. Tensor blk.89.attn_output.weight buffer type overriden to CUDA1
  1094. Tensor blk.89.attn_k_norm.weight buffer type overriden to CUDA1
  1095. Tensor blk.89.attn_q_norm.weight buffer type overriden to CUDA1
  1096. Tensor blk.89.ffn_norm.weight buffer type overriden to CUDA1
  1097. Tensor blk.89.ffn_gate_inp.weight buffer type overriden to CUDA1
  1098. Tensor blk.89.ffn_gate_exps.weight buffer type overriden to CPU
  1099. Tensor blk.89.ffn_down_exps.weight buffer type overriden to CPU
  1100. Tensor blk.89.ffn_up_exps.weight buffer type overriden to CPU
  1101. Tensor blk.90.attn_norm.weight buffer type overriden to CUDA1
  1102. Tensor blk.90.attn_q.weight buffer type overriden to CUDA1
  1103. Tensor blk.90.attn_k.weight buffer type overriden to CUDA1
  1104. Tensor blk.90.attn_v.weight buffer type overriden to CUDA1
  1105. Tensor blk.90.attn_output.weight buffer type overriden to CUDA1
  1106. Tensor blk.90.attn_k_norm.weight buffer type overriden to CUDA1
  1107. Tensor blk.90.attn_q_norm.weight buffer type overriden to CUDA1
  1108. Tensor blk.90.ffn_norm.weight buffer type overriden to CUDA1
  1109. Tensor blk.90.ffn_gate_inp.weight buffer type overriden to CUDA1
  1110. Tensor blk.90.ffn_gate_exps.weight buffer type overriden to CPU
  1111. Tensor blk.90.ffn_down_exps.weight buffer type overriden to CPU
  1112. Tensor blk.90.ffn_up_exps.weight buffer type overriden to CPU
  1113. Tensor blk.91.attn_norm.weight buffer type overriden to CUDA1
  1114. Tensor blk.91.attn_q.weight buffer type overriden to CUDA1
  1115. Tensor blk.91.attn_k.weight buffer type overriden to CUDA1
  1116. Tensor blk.91.attn_v.weight buffer type overriden to CUDA1
  1117. Tensor blk.91.attn_output.weight buffer type overriden to CUDA1
  1118. Tensor blk.91.attn_k_norm.weight buffer type overriden to CUDA1
  1119. Tensor blk.91.attn_q_norm.weight buffer type overriden to CUDA1
  1120. Tensor blk.91.ffn_norm.weight buffer type overriden to CUDA1
  1121. Tensor blk.91.ffn_gate_inp.weight buffer type overriden to CUDA1
  1122. Tensor blk.91.ffn_gate_exps.weight buffer type overriden to CPU
  1123. Tensor blk.91.ffn_down_exps.weight buffer type overriden to CPU
  1124. Tensor blk.91.ffn_up_exps.weight buffer type overriden to CPU
  1125. Tensor blk.92.attn_norm.weight buffer type overriden to CUDA1
  1126. Tensor blk.92.attn_q.weight buffer type overriden to CUDA1
  1127. Tensor blk.92.attn_k.weight buffer type overriden to CUDA1
  1128. Tensor blk.92.attn_v.weight buffer type overriden to CUDA1
  1129. Tensor blk.92.attn_output.weight buffer type overriden to CUDA1
  1130. Tensor blk.92.attn_k_norm.weight buffer type overriden to CUDA1
  1131. Tensor blk.92.attn_q_norm.weight buffer type overriden to CUDA1
  1132. Tensor blk.92.ffn_norm.weight buffer type overriden to CUDA1
  1133. Tensor blk.92.ffn_gate_inp.weight buffer type overriden to CUDA1
  1134. Tensor blk.92.ffn_gate_exps.weight buffer type overriden to CPU
  1135. Tensor blk.92.ffn_down_exps.weight buffer type overriden to CPU
  1136. Tensor blk.92.ffn_up_exps.weight buffer type overriden to CPU
  1137. Tensor blk.93.attn_norm.weight buffer type overriden to CUDA1
  1138. Tensor blk.93.attn_q.weight buffer type overriden to CUDA1
  1139. Tensor blk.93.attn_k.weight buffer type overriden to CUDA1
  1140. Tensor blk.93.attn_v.weight buffer type overriden to CUDA1
  1141. Tensor blk.93.attn_output.weight buffer type overriden to CUDA1
  1142. Tensor blk.93.attn_k_norm.weight buffer type overriden to CUDA1
  1143. Tensor blk.93.attn_q_norm.weight buffer type overriden to CUDA1
  1144. Tensor blk.93.ffn_norm.weight buffer type overriden to CUDA1
  1145. Tensor blk.93.ffn_gate_inp.weight buffer type overriden to CUDA1
  1146. Tensor blk.93.ffn_gate_exps.weight buffer type overriden to CPU
  1147. Tensor blk.93.ffn_down_exps.weight buffer type overriden to CPU
  1148. Tensor blk.93.ffn_up_exps.weight buffer type overriden to CPU
  1149. llm_load_tensors: offloading 94 repeating layers to GPU
  1150. llm_load_tensors: offloaded 94/95 layers to GPU
  1151. llm_load_tensors: CPU buffer size = 24919.24 MiB
  1152. llm_load_tensors: CPU buffer size = 37141.03 MiB
  1153. llm_load_tensors: CPU buffer size = 35082.59 MiB
  1154. llm_load_tensors: CPU buffer size = 630.59 MiB
  1155. llm_load_tensors: CPU buffer size = 1722.64 MiB
  1156. llm_load_tensors: CUDA0 buffer size = 13662.20 MiB
  1157. llm_load_tensors: CUDA1 buffer size = 13662.20 MiB
  1158. ....................................................................................................
  1159. llama_new_context_with_model: n_ctx = 256
  1160. llama_new_context_with_model: n_batch = 256
  1161. llama_new_context_with_model: n_ubatch = 256
  1162. llama_new_context_with_model: flash_attn = 1
  1163. llama_new_context_with_model: mla_attn = 0
  1164. llama_new_context_with_model: attn_max_b = 0
  1165. llama_new_context_with_model: fused_moe = 1
  1166. llama_new_context_with_model: ser = -1, 0
  1167. llama_new_context_with_model: freq_base = 1000000.0
  1168. llama_new_context_with_model: freq_scale = 1
  1169. llama_kv_cache_init: CUDA0 KV buffer size = 23.50 MiB
  1170. llama_kv_cache_init: CUDA1 KV buffer size = 23.50 MiB
  1171. llama_new_context_with_model: KV self size = 47.00 MiB, K (f16): 23.50 MiB, V (f16): 23.50 MiB
  1172. llama_new_context_with_model: CUDA_Host output buffer size = 0.58 MiB
  1173. llama_new_context_with_model: CUDA0 compute buffer size = 786.98 MiB
  1174. llama_new_context_with_model: CUDA1 compute buffer size = 60.13 MiB
  1175. llama_new_context_with_model: CUDA_Host compute buffer size = 4.25 MiB
  1176. llama_new_context_with_model: graph nodes = 3672
  1177. llama_new_context_with_model: graph splits = 189
  1178. | qwen3moe ?B IQ3_K - 3.4325 bpw | 106.83 GiB | 235.09 B | CUDA | 94 | 16 | 1 | 1 | pp64 | 2.35 ± 0.73 |
  1179.  
  1180. llama_print_timings: load time = 55532.55 ms
  1181. llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
  1182. llama_print_timings: prompt eval time = 152150.46 ms / 320 tokens ( 475.47 ms per token, 2.10 tokens per second)
  1183. llama_print_timings: eval time = 523.36 ms / 1 runs ( 523.36 ms per token, 1.91 tokens per second)
  1184. llama_print_timings: total time = 207687.90 ms / 321 tokens
  1185. llama_new_context_with_model: n_ctx = 256
  1186. llama_new_context_with_model: n_batch = 256
  1187. llama_new_context_with_model: n_ubatch = 256
  1188. llama_new_context_with_model: flash_attn = 1
  1189. llama_new_context_with_model: mla_attn = 0
  1190. llama_new_context_with_model: attn_max_b = 0
  1191. llama_new_context_with_model: fused_moe = 1
  1192. llama_new_context_with_model: ser = -1, 0
  1193. llama_new_context_with_model: freq_base = 1000000.0
  1194. llama_new_context_with_model: freq_scale = 1
  1195. llama_kv_cache_init: CUDA0 KV buffer size = 23.50 MiB
  1196. llama_kv_cache_init: CUDA1 KV buffer size = 23.50 MiB
  1197. llama_new_context_with_model: KV self size = 47.00 MiB, K (f16): 23.50 MiB, V (f16): 23.50 MiB
  1198. llama_new_context_with_model: CUDA_Host output buffer size = 0.58 MiB
  1199. llama_new_context_with_model: CUDA0 compute buffer size = 786.98 MiB
  1200. llama_new_context_with_model: CUDA1 compute buffer size = 60.13 MiB
  1201. llama_new_context_with_model: CUDA_Host compute buffer size = 4.25 MiB
  1202. llama_new_context_with_model: graph nodes = 3672
  1203. llama_new_context_with_model: graph splits = 189
  1204. | qwen3moe ?B IQ3_K - 3.4325 bpw | 106.83 GiB | 235.09 B | CUDA | 94 | 16 | 1 | 1 | tg32 | 3.76 ± 1.08 |
  1205.  
  1206. llama_print_timings: load time = 209061.32 ms
  1207. llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
  1208. llama_print_timings: prompt eval time = 0.00 ms / 0 tokens ( -nan ms per token, -nan tokens per second)
  1209. llama_print_timings: eval time = 48630.51 ms / 161 runs ( 302.05 ms per token, 3.31 tokens per second)
  1210. llama_print_timings: total time = 256431.90 ms / 161 tokens
  1211.  
  1212. build: 0435b68e (3693)
  1213.  
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!