API tools faq
paste
Lissanro

ik_llama.cpp debug log, part 1

Lissanro
May 23rd, 2025
68
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 34.63 KB | None | 0 0
raw download clone embed print report
  1. ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
  2. ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
  3. ggml_cuda_init: found 4 CUDA devices:
  4. Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  5. Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  6. Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  7. Device 3: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  8. INFO [ main] build info | tid="137034227900416" timestamp=1747986541 build=3707 commit="a1c931c3"
  9. INFO [ main] system info | tid="137034227900416" timestamp=1747986541 n_threads=64 n_threads_batch=-1 total_threads=128 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | "
  10. llama_model_loader: loaded meta data with 59 key-value pairs and 1147 tensors from /mnt/neuro/models/DeepSeek-R1T-Chimera-256x21B-IQ4_K-163840seq/DeepSeek-R1T-Chimera-256x21B-IQ4_K-163840seq.gguf (version GGUF V3 (latest))
  11. llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
  12. llama_model_loader: - kv 0: general.architecture str = deepseek2
  13. llama_model_loader: - kv 1: general.type str = model
  14. llama_model_loader: - kv 2: general.name str = DeepSeek R1T Chimera 163840seq
  15. llama_model_loader: - kv 3: general.finetune str = 163840seq
  16. llama_model_loader: - kv 4: general.basename str = DeepSeek-R1T-Chimera
  17. llama_model_loader: - kv 5: general.size_label str = 256x21B
  18. llama_model_loader: - kv 6: general.license str = mit
  19. llama_model_loader: - kv 7: general.base_model.count u32 = 2
  20. llama_model_loader: - kv 8: general.base_model.0.name str = DeepSeek V3 0324
  21. llama_model_loader: - kv 9: general.base_model.0.version str = V3-0324
  22. llama_model_loader: - kv 10: general.base_model.0.organization str = Deepseek Ai
  23. llama_model_loader: - kv 11: general.base_model.0.repo_url str = https://huggingface.co/deepseek-ai/De...
  24. llama_model_loader: - kv 12: general.base_model.1.name str = DeepSeek R1
  25. llama_model_loader: - kv 13: general.base_model.1.organization str = Deepseek Ai
  26. llama_model_loader: - kv 14: general.base_model.1.repo_url str = https://huggingface.co/deepseek-ai/De...
  27. llama_model_loader: - kv 15: general.tags arr[str,1] = ["text-generation"]
  28. llama_model_loader: - kv 16: deepseek2.block_count u32 = 61
  29. llama_model_loader: - kv 17: deepseek2.context_length u32 = 163840
  30. llama_model_loader: - kv 18: deepseek2.embedding_length u32 = 7168
  31. llama_model_loader: - kv 19: deepseek2.feed_forward_length u32 = 18432
  32. llama_model_loader: - kv 20: deepseek2.attention.head_count u32 = 128
  33. llama_model_loader: - kv 21: deepseek2.attention.head_count_kv u32 = 128
  34. llama_model_loader: - kv 22: deepseek2.rope.freq_base f32 = 10000.000000
  35. llama_model_loader: - kv 23: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001
  36. llama_model_loader: - kv 24: deepseek2.expert_used_count u32 = 8
  37. llama_model_loader: - kv 25: general.file_type u32 = 140
  38. llama_model_loader: - kv 26: deepseek2.leading_dense_block_count u32 = 3
  39. llama_model_loader: - kv 27: deepseek2.vocab_size u32 = 129280
  40. llama_model_loader: - kv 28: deepseek2.attention.q_lora_rank u32 = 1536
  41. llama_model_loader: - kv 29: deepseek2.attention.kv_lora_rank u32 = 512
  42. llama_model_loader: - kv 30: deepseek2.attention.key_length u32 = 192
  43. llama_model_loader: - kv 31: deepseek2.attention.value_length u32 = 128
  44. llama_model_loader: - kv 32: deepseek2.expert_feed_forward_length u32 = 2048
  45. llama_model_loader: - kv 33: deepseek2.expert_count u32 = 256
  46. llama_model_loader: - kv 34: deepseek2.expert_shared_count u32 = 1
  47. llama_model_loader: - kv 35: deepseek2.expert_weights_scale f32 = 2.500000
  48. llama_model_loader: - kv 36: deepseek2.expert_weights_norm bool = true
  49. llama_model_loader: - kv 37: deepseek2.expert_gating_func u32 = 2
  50. llama_model_loader: - kv 38: deepseek2.rope.dimension_count u32 = 64
  51. llama_model_loader: - kv 39: deepseek2.rope.scaling.type str = yarn
  52. llama_model_loader: - kv 40: deepseek2.rope.scaling.factor f32 = 40.000000
  53. llama_model_loader: - kv 41: deepseek2.rope.scaling.original_context_length u32 = 4096
  54. llama_model_loader: - kv 42: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000
  55. llama_model_loader: - kv 43: tokenizer.ggml.model str = gpt2
  56. llama_model_loader: - kv 44: tokenizer.ggml.pre str = deepseek-v3
  57. llama_model_loader: - kv 45: tokenizer.ggml.tokens arr[str,129280] = ["<｜begin▁of▁sentence｜>", "<�...
  58. llama_model_loader: - kv 46: tokenizer.ggml.token_type arr[i32,129280] = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
  59. llama_model_loader: - kv 47: tokenizer.ggml.merges arr[str,127741] = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
  60. llama_model_loader: - kv 48: tokenizer.ggml.bos_token_id u32 = 0
  61. llama_model_loader: - kv 49: tokenizer.ggml.eos_token_id u32 = 1
  62. llama_model_loader: - kv 50: tokenizer.ggml.padding_token_id u32 = 1
  63. llama_model_loader: - kv 51: tokenizer.ggml.add_bos_token bool = true
  64. llama_model_loader: - kv 52: tokenizer.ggml.add_eos_token bool = false
  65. llama_model_loader: - kv 53: tokenizer.chat_template str = {% if not add_generation_prompt is de...
  66. llama_model_loader: - kv 54: general.quantization_version u32 = 2
  67. llama_model_loader: - kv 55: quantize.imatrix.file str = /home/lissanro/pkgs/imatrix/DeepSeek-...
  68. llama_model_loader: - kv 56: quantize.imatrix.dataset str = /home/lissanro/pkgs/imatrix/all.txt
  69. llama_model_loader: - kv 57: quantize.imatrix.entries_count i32 = 543
  70. llama_model_loader: - kv 58: quantize.imatrix.chunks_count i32 = 3660
  71. llama_model_loader: - type f32: 361 tensors
  72. llama_model_loader: - type q8_0: 305 tensors
  73. llama_model_loader: - type q5_K: 61 tensors
  74. llama_model_loader: - type q6_K: 1 tensors
  75. llama_model_loader: - type iq4_k: 419 tensors
  76. llm_load_vocab: special tokens cache size = 818
  77. llm_load_vocab: token to piece cache size = 0.8223 MB
  78. llm_load_print_meta: format = GGUF V3 (latest)
  79. llm_load_print_meta: arch = deepseek2
  80. llm_load_print_meta: vocab type = BPE
  81. llm_load_print_meta: n_vocab = 129280
  82. llm_load_print_meta: n_merges = 127741
  83. llm_load_print_meta: vocab_only = 0
  84. llm_load_print_meta: n_ctx_train = 163840
  85. llm_load_print_meta: n_embd = 7168
  86. llm_load_print_meta: n_layer = 61
  87. llm_load_print_meta: n_head = 128
  88. llm_load_print_meta: n_head_kv = 128
  89. llm_load_print_meta: n_rot = 64
  90. llm_load_print_meta: n_swa = 0
  91. llm_load_print_meta: n_swa_pattern = 1
  92. llm_load_print_meta: n_embd_head_k = 192
  93. llm_load_print_meta: n_embd_head_v = 128
  94. llm_load_print_meta: n_gqa = 1
  95. llm_load_print_meta: n_embd_k_gqa = 24576
  96. llm_load_print_meta: n_embd_v_gqa = 16384
  97. llm_load_print_meta: f_norm_eps = 0.0e+00
  98. llm_load_print_meta: f_norm_rms_eps = 1.0e-06
  99. llm_load_print_meta: f_clamp_kqv = 0.0e+00
  100. llm_load_print_meta: f_max_alibi_bias = 0.0e+00
  101. llm_load_print_meta: f_logit_scale = 0.0e+00
  102. llm_load_print_meta: n_ff = 18432
  103. llm_load_print_meta: n_expert = 256
  104. llm_load_print_meta: n_expert_used = 8
  105. llm_load_print_meta: causal attn = 1
  106. llm_load_print_meta: pooling type = 0
  107. llm_load_print_meta: rope type = 0
  108. llm_load_print_meta: rope scaling = yarn
  109. llm_load_print_meta: freq_base_train = 10000.0
  110. llm_load_print_meta: freq_scale_train = 0.025
  111. llm_load_print_meta: n_ctx_orig_yarn = 4096
  112. llm_load_print_meta: rope_finetuned = unknown
  113. llm_load_print_meta: ssm_d_conv = 0
  114. llm_load_print_meta: ssm_d_inner = 0
  115. llm_load_print_meta: ssm_d_state = 0
  116. llm_load_print_meta: ssm_dt_rank = 0
  117. llm_load_print_meta: model type = 671B
  118. llm_load_print_meta: model ftype = IQ4_K - 4.5 bpw
  119. llm_load_print_meta: model params = 672.050 B
  120. llm_load_print_meta: model size = 355.684 GiB (4.546 BPW)
  121. llm_load_print_meta: repeating layers = 354.490 GiB (4.544 BPW, 670.196 B parameters)
  122. llm_load_print_meta: general.name = DeepSeek R1T Chimera 163840seq
  123. llm_load_print_meta: BOS token = 0 '<｜begin▁of▁sentence｜>'
  124. llm_load_print_meta: EOS token = 1 '<｜end▁of▁sentence｜>'
  125. llm_load_print_meta: PAD token = 1 '<｜end▁of▁sentence｜>'
  126. llm_load_print_meta: LF token = 131 'Ä'
  127. llm_load_print_meta: max token length = 256
  128. llm_load_print_meta: n_layer_dense_lead = 3
  129. llm_load_print_meta: n_lora_q = 1536
  130. llm_load_print_meta: n_lora_kv = 512
  131. llm_load_print_meta: n_ff_exp = 2048
  132. llm_load_print_meta: n_expert_shared = 1
  133. llm_load_print_meta: expert_weights_scale = 2.5
  134. llm_load_print_meta: expert_weights_norm = 1
  135. llm_load_print_meta: expert_gating_func = sigmoid
  136. llm_load_print_meta: rope_yarn_log_mul = 0.1000
  137. llm_load_tensors: ggml ctx size = 2.34 MiB
  138. Tensor blk.3.ffn_gate_exps.weight buffer type overriden to CUDA0
  139. Tensor blk.3.ffn_down_exps.weight buffer type overriden to CUDA0
  140. Tensor blk.3.ffn_up_exps.weight buffer type overriden to CUDA0
  141. Tensor blk.4.ffn_gate_exps.weight buffer type overriden to CUDA1
  142. Tensor blk.4.ffn_down_exps.weight buffer type overriden to CUDA1
  143. Tensor blk.4.ffn_up_exps.weight buffer type overriden to CUDA1
  144. Tensor blk.5.ffn_gate_exps.weight buffer type overriden to CUDA2
  145. Tensor blk.5.ffn_down_exps.weight buffer type overriden to CUDA2
  146. Tensor blk.5.ffn_up_exps.weight buffer type overriden to CUDA2
  147. Tensor blk.6.ffn_gate_exps.weight buffer type overriden to CUDA3
  148. Tensor blk.6.ffn_down_exps.weight buffer type overriden to CUDA3
  149. Tensor blk.6.ffn_up_exps.weight buffer type overriden to CUDA3
  150. Tensor blk.7.ffn_gate_exps.weight buffer type overriden to CPU
  151. Tensor blk.7.ffn_down_exps.weight buffer type overriden to CPU
  152. Tensor blk.7.ffn_up_exps.weight buffer type overriden to CPU
  153. Tensor blk.8.ffn_gate_exps.weight buffer type overriden to CPU
  154. Tensor blk.8.ffn_down_exps.weight buffer type overriden to CPU
  155. Tensor blk.8.ffn_up_exps.weight buffer type overriden to CPU
  156. Tensor blk.9.ffn_gate_exps.weight buffer type overriden to CPU
  157. Tensor blk.9.ffn_down_exps.weight buffer type overriden to CPU
  158. Tensor blk.9.ffn_up_exps.weight buffer type overriden to CPU
  159. Tensor blk.10.ffn_gate_exps.weight buffer type overriden to CPU
  160. Tensor blk.10.ffn_down_exps.weight buffer type overriden to CPU
  161. Tensor blk.10.ffn_up_exps.weight buffer type overriden to CPU
  162. Tensor blk.11.ffn_gate_exps.weight buffer type overriden to CPU
  163. Tensor blk.11.ffn_down_exps.weight buffer type overriden to CPU
  164. Tensor blk.11.ffn_up_exps.weight buffer type overriden to CPU
  165. Tensor blk.12.ffn_gate_exps.weight buffer type overriden to CPU
  166. Tensor blk.12.ffn_down_exps.weight buffer type overriden to CPU
  167. Tensor blk.12.ffn_up_exps.weight buffer type overriden to CPU
  168. Tensor blk.13.ffn_gate_exps.weight buffer type overriden to CPU
  169. Tensor blk.13.ffn_down_exps.weight buffer type overriden to CPU
  170. Tensor blk.13.ffn_up_exps.weight buffer type overriden to CPU
  171. Tensor blk.14.ffn_gate_exps.weight buffer type overriden to CPU
  172. Tensor blk.14.ffn_down_exps.weight buffer type overriden to CPU
  173. Tensor blk.14.ffn_up_exps.weight buffer type overriden to CPU
  174. Tensor blk.15.ffn_gate_exps.weight buffer type overriden to CPU
  175. Tensor blk.15.ffn_down_exps.weight buffer type overriden to CPU
  176. Tensor blk.15.ffn_up_exps.weight buffer type overriden to CPU
  177. Tensor blk.16.ffn_gate_exps.weight buffer type overriden to CPU
  178. Tensor blk.16.ffn_down_exps.weight buffer type overriden to CPU
  179. Tensor blk.16.ffn_up_exps.weight buffer type overriden to CPU
  180. Tensor blk.17.ffn_gate_exps.weight buffer type overriden to CPU
  181. Tensor blk.17.ffn_down_exps.weight buffer type overriden to CPU
  182. Tensor blk.17.ffn_up_exps.weight buffer type overriden to CPU
  183. Tensor blk.18.ffn_gate_exps.weight buffer type overriden to CPU
  184. Tensor blk.18.ffn_down_exps.weight buffer type overriden to CPU
  185. Tensor blk.18.ffn_up_exps.weight buffer type overriden to CPU
  186. Tensor blk.19.ffn_gate_exps.weight buffer type overriden to CPU
  187. Tensor blk.19.ffn_down_exps.weight buffer type overriden to CPU
  188. Tensor blk.19.ffn_up_exps.weight buffer type overriden to CPU
  189. Tensor blk.20.ffn_gate_exps.weight buffer type overriden to CPU
  190. Tensor blk.20.ffn_down_exps.weight buffer type overriden to CPU
  191. Tensor blk.20.ffn_up_exps.weight buffer type overriden to CPU
  192. Tensor blk.21.ffn_gate_exps.weight buffer type overriden to CPU
  193. Tensor blk.21.ffn_down_exps.weight buffer type overriden to CPU
  194. Tensor blk.21.ffn_up_exps.weight buffer type overriden to CPU
  195. Tensor blk.22.ffn_gate_exps.weight buffer type overriden to CPU
  196. Tensor blk.22.ffn_down_exps.weight buffer type overriden to CPU
  197. Tensor blk.22.ffn_up_exps.weight buffer type overriden to CPU
  198. Tensor blk.23.ffn_gate_exps.weight buffer type overriden to CPU
  199. Tensor blk.23.ffn_down_exps.weight buffer type overriden to CPU
  200. Tensor blk.23.ffn_up_exps.weight buffer type overriden to CPU
  201. Tensor blk.24.ffn_gate_exps.weight buffer type overriden to CPU
  202. Tensor blk.24.ffn_down_exps.weight buffer type overriden to CPU
  203. Tensor blk.24.ffn_up_exps.weight buffer type overriden to CPU
  204. Tensor blk.25.ffn_gate_exps.weight buffer type overriden to CPU
  205. Tensor blk.25.ffn_down_exps.weight buffer type overriden to CPU
  206. Tensor blk.25.ffn_up_exps.weight buffer type overriden to CPU
  207. Tensor blk.26.ffn_gate_exps.weight buffer type overriden to CPU
  208. Tensor blk.26.ffn_down_exps.weight buffer type overriden to CPU
  209. Tensor blk.26.ffn_up_exps.weight buffer type overriden to CPU
  210. Tensor blk.27.ffn_gate_exps.weight buffer type overriden to CPU
  211. Tensor blk.27.ffn_down_exps.weight buffer type overriden to CPU
  212. Tensor blk.27.ffn_up_exps.weight buffer type overriden to CPU
  213. Tensor blk.28.ffn_gate_exps.weight buffer type overriden to CPU
  214. Tensor blk.28.ffn_down_exps.weight buffer type overriden to CPU
  215. Tensor blk.28.ffn_up_exps.weight buffer type overriden to CPU
  216. Tensor blk.29.ffn_gate_exps.weight buffer type overriden to CPU
  217. Tensor blk.29.ffn_down_exps.weight buffer type overriden to CPU
  218. Tensor blk.29.ffn_up_exps.weight buffer type overriden to CPU
  219. Tensor blk.30.ffn_gate_exps.weight buffer type overriden to CPU
  220. Tensor blk.30.ffn_down_exps.weight buffer type overriden to CPU
  221. Tensor blk.30.ffn_up_exps.weight buffer type overriden to CPU
  222. Tensor blk.31.ffn_gate_exps.weight buffer type overriden to CPU
  223. Tensor blk.31.ffn_down_exps.weight buffer type overriden to CPU
  224. Tensor blk.31.ffn_up_exps.weight buffer type overriden to CPU
  225. Tensor blk.32.ffn_gate_exps.weight buffer type overriden to CPU
  226. Tensor blk.32.ffn_down_exps.weight buffer type overriden to CPU
  227. Tensor blk.32.ffn_up_exps.weight buffer type overriden to CPU
  228. Tensor blk.33.ffn_gate_exps.weight buffer type overriden to CPU
  229. Tensor blk.33.ffn_down_exps.weight buffer type overriden to CPU
  230. Tensor blk.33.ffn_up_exps.weight buffer type overriden to CPU
  231. Tensor blk.34.ffn_gate_exps.weight buffer type overriden to CPU
  232. Tensor blk.34.ffn_down_exps.weight buffer type overriden to CPU
  233. Tensor blk.34.ffn_up_exps.weight buffer type overriden to CPU
  234. Tensor blk.35.ffn_gate_exps.weight buffer type overriden to CPU
  235. Tensor blk.35.ffn_down_exps.weight buffer type overriden to CPU
  236. Tensor blk.35.ffn_up_exps.weight buffer type overriden to CPU
  237. Tensor blk.36.ffn_gate_exps.weight buffer type overriden to CPU
  238. Tensor blk.36.ffn_down_exps.weight buffer type overriden to CPU
  239. Tensor blk.36.ffn_up_exps.weight buffer type overriden to CPU
  240. Tensor blk.37.ffn_gate_exps.weight buffer type overriden to CPU
  241. Tensor blk.37.ffn_down_exps.weight buffer type overriden to CPU
  242. Tensor blk.37.ffn_up_exps.weight buffer type overriden to CPU
  243. Tensor blk.38.ffn_gate_exps.weight buffer type overriden to CPU
  244. Tensor blk.38.ffn_down_exps.weight buffer type overriden to CPU
  245. Tensor blk.38.ffn_up_exps.weight buffer type overriden to CPU
  246. Tensor blk.39.ffn_gate_exps.weight buffer type overriden to CPU
  247. Tensor blk.39.ffn_down_exps.weight buffer type overriden to CPU
  248. Tensor blk.39.ffn_up_exps.weight buffer type overriden to CPU
  249. Tensor blk.40.ffn_gate_exps.weight buffer type overriden to CPU
  250. Tensor blk.40.ffn_down_exps.weight buffer type overriden to CPU
  251. Tensor blk.40.ffn_up_exps.weight buffer type overriden to CPU
  252. Tensor blk.41.ffn_gate_exps.weight buffer type overriden to CPU
  253. Tensor blk.41.ffn_down_exps.weight buffer type overriden to CPU
  254. Tensor blk.41.ffn_up_exps.weight buffer type overriden to CPU
  255. Tensor blk.42.ffn_gate_exps.weight buffer type overriden to CPU
  256. Tensor blk.42.ffn_down_exps.weight buffer type overriden to CPU
  257. Tensor blk.42.ffn_up_exps.weight buffer type overriden to CPU
  258. Tensor blk.43.ffn_gate_exps.weight buffer type overriden to CPU
  259. Tensor blk.43.ffn_down_exps.weight buffer type overriden to CPU
  260. Tensor blk.43.ffn_up_exps.weight buffer type overriden to CPU
  261. Tensor blk.44.ffn_gate_exps.weight buffer type overriden to CPU
  262. Tensor blk.44.ffn_down_exps.weight buffer type overriden to CPU
  263. Tensor blk.44.ffn_up_exps.weight buffer type overriden to CPU
  264. Tensor blk.45.ffn_gate_exps.weight buffer type overriden to CPU
  265. Tensor blk.45.ffn_down_exps.weight buffer type overriden to CPU
  266. Tensor blk.45.ffn_up_exps.weight buffer type overriden to CPU
  267. Tensor blk.46.ffn_gate_exps.weight buffer type overriden to CPU
  268. Tensor blk.46.ffn_down_exps.weight buffer type overriden to CPU
  269. Tensor blk.46.ffn_up_exps.weight buffer type overriden to CPU
  270. Tensor blk.47.ffn_gate_exps.weight buffer type overriden to CPU
  271. Tensor blk.47.ffn_down_exps.weight buffer type overriden to CPU
  272. Tensor blk.47.ffn_up_exps.weight buffer type overriden to CPU
  273. Tensor blk.48.ffn_gate_exps.weight buffer type overriden to CPU
  274. Tensor blk.48.ffn_down_exps.weight buffer type overriden to CPU
  275. Tensor blk.48.ffn_up_exps.weight buffer type overriden to CPU
  276. Tensor blk.49.ffn_gate_exps.weight buffer type overriden to CPU
  277. Tensor blk.49.ffn_down_exps.weight buffer type overriden to CPU
  278. Tensor blk.49.ffn_up_exps.weight buffer type overriden to CPU
  279. Tensor blk.50.ffn_gate_exps.weight buffer type overriden to CPU
  280. Tensor blk.50.ffn_down_exps.weight buffer type overriden to CPU
  281. Tensor blk.50.ffn_up_exps.weight buffer type overriden to CPU
  282. Tensor blk.51.ffn_gate_exps.weight buffer type overriden to CPU
  283. Tensor blk.51.ffn_down_exps.weight buffer type overriden to CPU
  284. Tensor blk.51.ffn_up_exps.weight buffer type overriden to CPU
  285. Tensor blk.52.ffn_gate_exps.weight buffer type overriden to CPU
  286. Tensor blk.52.ffn_down_exps.weight buffer type overriden to CPU
  287. Tensor blk.52.ffn_up_exps.weight buffer type overriden to CPU
  288. Tensor blk.53.ffn_gate_exps.weight buffer type overriden to CPU
  289. Tensor blk.53.ffn_down_exps.weight buffer type overriden to CPU
  290. Tensor blk.53.ffn_up_exps.weight buffer type overriden to CPU
  291. Tensor blk.54.ffn_gate_exps.weight buffer type overriden to CPU
  292. Tensor blk.54.ffn_down_exps.weight buffer type overriden to CPU
  293. Tensor blk.54.ffn_up_exps.weight buffer type overriden to CPU
  294. Tensor blk.55.ffn_gate_exps.weight buffer type overriden to CPU
  295. Tensor blk.55.ffn_down_exps.weight buffer type overriden to CPU
  296. Tensor blk.55.ffn_up_exps.weight buffer type overriden to CPU
  297. Tensor blk.56.ffn_gate_exps.weight buffer type overriden to CPU
  298. Tensor blk.56.ffn_down_exps.weight buffer type overriden to CPU
  299. Tensor blk.56.ffn_up_exps.weight buffer type overriden to CPU
  300. Tensor blk.57.ffn_gate_exps.weight buffer type overriden to CPU
  301. Tensor blk.57.ffn_down_exps.weight buffer type overriden to CPU
  302. Tensor blk.57.ffn_up_exps.weight buffer type overriden to CPU
  303. Tensor blk.58.ffn_gate_exps.weight buffer type overriden to CPU
  304. Tensor blk.58.ffn_down_exps.weight buffer type overriden to CPU
  305. Tensor blk.58.ffn_up_exps.weight buffer type overriden to CPU
  306. Tensor blk.59.ffn_gate_exps.weight buffer type overriden to CPU
  307. Tensor blk.59.ffn_down_exps.weight buffer type overriden to CPU
  308. Tensor blk.59.ffn_up_exps.weight buffer type overriden to CPU
  309. Tensor blk.60.ffn_gate_exps.weight buffer type overriden to CPU
  310. Tensor blk.60.ffn_down_exps.weight buffer type overriden to CPU
  311. Tensor blk.60.ffn_up_exps.weight buffer type overriden to CPU
  312. llm_load_tensors: offloading 61 repeating layers to GPU
  313. llm_load_tensors: offloading non-repeating layers to GPU
  314. llm_load_tensors: offloaded 62/62 layers to GPU
  315. llm_load_tensors: CPU buffer size = 337454.70 MiB
  316. llm_load_tensors: CPU buffer size = 497.11 MiB
  317. llm_load_tensors: CUDA0 buffer size = 8506.79 MiB
  318. llm_load_tensors: CUDA1 buffer size = 8917.18 MiB
  319. llm_load_tensors: CUDA2 buffer size = 9682.30 MiB
  320. llm_load_tensors: CUDA3 buffer size = 10024.72 MiB
  321. ....................................................................................................
  322. llama_new_context_with_model: n_ctx = 102400
  323. llama_new_context_with_model: n_batch = 4096
  324. llama_new_context_with_model: n_ubatch = 4096
  325. llama_new_context_with_model: flash_attn = 1
  326. llama_new_context_with_model: mla_attn = 3
  327. llama_new_context_with_model: attn_max_b = 1024
  328. llama_new_context_with_model: fused_moe = 1
  329. llama_new_context_with_model: ser = -1, 0
  330. llama_new_context_with_model: freq_base = 10000.0
  331. llama_new_context_with_model: freq_scale = 0.025
  332. llama_kv_cache_init: CUDA0 KV buffer size = 597.66 MiB
  333. llama_kv_cache_init: CUDA1 KV buffer size = 896.49 MiB
  334. llama_kv_cache_init: CUDA2 KV buffer size = 1135.56 MiB
  335. llama_kv_cache_init: CUDA3 KV buffer size = 1016.02 MiB
  336. llama_new_context_with_model: KV self size = 3645.70 MiB, c^KV (q8_0): 3645.70 MiB, kv^T: not used
  337. llama_new_context_with_model: CUDA_Host output buffer size = 0.99 MiB
  338. llama_new_context_with_model: pipeline parallelism enabled (n_copies=1)
  339. llama_new_context_with_model: CUDA0 compute buffer size = 11732.02 MiB
  340. llama_new_context_with_model: CUDA1 compute buffer size = 10332.02 MiB
  341. llama_new_context_with_model: CUDA2 compute buffer size = 10332.02 MiB
  342. llama_new_context_with_model: CUDA3 compute buffer size = 10332.03 MiB
  343. llama_new_context_with_model: CUDA_Host compute buffer size = 1712.05 MiB
  344. llama_new_context_with_model: graph nodes = 13613
  345. llama_new_context_with_model: graph splits = 170
  346. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 1 without access enabled
  347. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 1 without access enabled
  348. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 1 without access enabled
  349. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 1 without access enabled
  350. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 1 without access enabled
  351. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 1 without access enabled
  352. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 1 to device 0 without access enabled
  353. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 1 to device 0 without access enabled
  354. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 2 without access enabled
  355. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 2 without access enabled
  356. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 2 without access enabled
  357. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 2 without access enabled
  358. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 2 without access enabled
  359. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 2 without access enabled
  360. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 2 to device 0 without access enabled
  361. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 2 to device 0 without access enabled
  362. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 3 without access enabled
  363. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 3 without access enabled
  364. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 3 without access enabled
  365. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 3 without access enabled
  366. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 3 without access enabled
  367. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 3 without access enabled
  368. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 3 to device 0 without access enabled
  369. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 3 to device 0 without access enabled
  370. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 1 without access enabled
  371. ...
  372. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 3 without access enabled
  373. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 3 to device 0 without access enabled
  374. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 3 to device 0 without access enabled
  375. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 3 to device 0 without access enabled
  376. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 3 to device 0 without access enabled
  377. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 3 to device 0 without access enabled
  378. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 3 to device 0 without access enabled
  379. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 3 without access enabled
  380. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 3 without access enabled
  381. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 3 to device 0 without access enabled
  382. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 3 to device 0 without access enabled
  383. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 3 to device 0 without access enabled
  384. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 3 to device 0 without access enabled
  385. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 3 to device 0 without access enabled
  386. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 3 to device 0 without access enabled
  387. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 3 without access enabled
  388. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 3 without access enabled
  389. INFO [ init] initializing slots | tid="137034227900416" timestamp=1747986642 n_slots=1
  390. INFO [ init] new slot | tid="137034227900416" timestamp=1747986642 id_slot=0 n_ctx_slot=102400
  391. INFO [ main] model loaded | tid="137034227900416" timestamp=1747986642
  392. INFO [ main] chat template | tid="137034227900416" timestamp=1747986642 chat_example="You are a helpful assistant\n\n<｜User｜>Hello<｜Assistant｜>Hi there<｜end▁of▁sentence｜><｜User｜>How are you?<｜Assistant｜>" built_in=true
  393. INFO [ main] HTTP server listening | tid="137034227900416" timestamp=1747986642 n_threads_http="127" port="5000" hostname="0.0.0.0"
  394. INFO [ update_slots] all slots are idle | tid="137034227900416" timestamp=1747986642
  395. INFO [ launch_slot_with_task] slot is processing task | tid="137034227900416" timestamp=1747986667 id_slot=0 id_task=0
  396. INFO [ update_slots] kv cache rm [p0, end) | tid="137034227900416" timestamp=1747986667 id_slot=0 id_task=0 p0=0
  397. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 1 without access enabled
  398. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 1 without access enabled
  399. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 1 without access enabled
  400. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 1 without access enabled
  401. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 1 without access enabled
  402. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 1 without access enabled
  403. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 1 to device 0 without access enabled
  404. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 1 to device 0 without access enabled
  405. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 2 without access enabled
  406. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 2 without access enabled
  407. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 2 without access enabled
  408. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 2 without access enabled
  409. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 2 without access enabled
  410. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 2 without access enabled
  411. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 2 to device 0 without access enabled
  412. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 2 to device 0 without access enabled
  413. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 3 without access enabled
  414. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 3 without access enabled
  415. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 3 without access enabled
  416. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 3 without access enabled
  417. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 3 without access enabled
  418. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 3 without access enabled
  419. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 3 to device 0 without access enabled
  420. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 3 to device 0 without access enabled
  421. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 1 without access enabled
  422. ...
  423. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 3 without access enabled
  424. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 3 without access enabled
  425. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 3 without access enabled
  426. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 3 without access enabled
  427. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 3 without access enabled
  428. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 3 without access enabled
  429. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 3 to device 0 without access enabled
  430. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 3 to device 0 without access enabled
  431. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 1 without access enabled
  432. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 1 without access enabled
  433. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 1 without access enabled
  434. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 1 without access enabled
  435. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 1 to device 2 without access enabled
  436. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 1 to device 2 without access enabled
  437. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 2 without access enabled
  438. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 2 without access enabled
  439. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 2 to device 3 without access enabled
  440. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 2 to device 3 without access enabled
  441. ggml_backend_cuda_cpy_tensor_async: attempt to copy from device 0 to device 3 without access enabled
  442. ggml_backend_cuda_buffer_cpy_tensor: attempt to copy from device 0 to device 3 without access enabled
  443. INFO [ print_timings] prompt eval time = 29845.93 ms / 713 tokens ( 41.86 ms per token, 23.89 tokens per second) | tid="137034227900416" timestamp=1747986842 id_slot=0 id_task=0 t_prompt_processing=29845.931 n_prompt_tokens_processed=713 t_token=41.8596507713885 n_tokens_second=23.889353627467678
  444. INFO [ print_timings] generation eval time = 145064.81 ms / 1053 runs ( 137.76 ms per token, 7.26 tokens per second) | tid="137034227900416" timestamp=1747986842 id_slot=0 id_task=0 t_token_generation=145064.811 n_decoded=1053 t_token=137.76335327635326 n_tokens_second=7.25882447122204
  445. INFO [ print_timings] total time = 174910.74 ms | tid="137034227900416" timestamp=1747986842 id_slot=0 id_task=0 t_prompt_processing=29845.931 t_token_generation=145064.811 t_total=174910.742
  446. INFO [ update_slots] slot released | tid="137034227900416" timestamp=1747986842 id_slot=0 id_task=0 n_ctx=102400 n_past=1765 n_system_tokens=0 n_cache_tokens=1765 truncated=false
  447. INFO [ update_slots] all slots are idle | tid="137034227900416" timestamp=1747986842
  448. INFO [ log_server_request] request | tid="136631981682688" timestamp=1747986842 remote_addr="127.0.0.1" remote_port=49724 status=200 method="POST" path="/completion" params={}
  449. INFO [ update_slots] all slots are idle | tid="137034227900416" timestamp=1747986842
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!