API tools faq
paste
Guest User

Untitled

a guest
Mar 7th, 2026
27
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 16.10 KB | None | 0 0
raw download clone embed print report
  1. $ build/bin/llama-server -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL --ctx-size 16384 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 --parallel 1
  2. ggml_cuda_init: found 1 CUDA devices:
  3. Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
  4. common_download_file_single_online: no previous model file found /home/user/.cache/llama.cpp/unsloth_Qwen3.5-35B-A3B-GGUF_preset.ini
  5. common_download_file_single_online: HEAD failed, status: 404
  6. no remote preset found, skipping
  7. common_download_file_single_online: using cached file (same etag): /home/user/.cache/llama.cpp/unsloth_Qwen3.5-35B-A3B-GGUF_Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf
  8. common_download_file_single_online: using cached file (same etag): /home/user/.cache/llama.cpp/unsloth_Qwen3.5-35B-A3B-GGUF_mmproj-BF16.gguf
  9. build: 8233 (c5a778891) with GNU 11.4.0 for Linux x86_64
  10. system info: n_threads = 8, n_threads_batch = 8, total_threads = 16
  11.  
  12. system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CUDA : ARCHS = 500,610,700,750,800,860,890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
  13.  
  14. Running without SSL
  15. init: using 15 threads for HTTP server
  16. start: binding port with default address family
  17. main: loading model
  18. srv load_model: loading model '/home/user/.cache/llama.cpp/unsloth_Qwen3.5-35B-A3B-GGUF_Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf'
  19. common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
  20. llama_params_fit_impl: projected to use 21561 MiB of device memory vs. 22924 MiB of free device memory
  21. llama_params_fit_impl: will leave 1362 >= 1024 MiB of free device memory, no changes needed
  22. llama_params_fit: successfully fit params to free device memory
  23. llama_params_fit: fitting params to free memory took 0.68 seconds
  24. llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4090) (0000:07:00.0) - 22988 MiB free
  25. llama_model_loader: loaded meta data with 52 key-value pairs and 733 tensors from /home/user/.cache/llama.cpp/unsloth_Qwen3.5-35B-A3B-GGUF_Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf (version GGUF V3 (latest))
  26. llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
  27. llama_model_loader: - kv 0: general.architecture str = qwen35moe
  28. llama_model_loader: - kv 1: general.type str = model
  29. llama_model_loader: - kv 2: general.sampling.top_k i32 = 20
  30. llama_model_loader: - kv 3: general.sampling.top_p f32 = 0.950000
  31. llama_model_loader: - kv 4: general.sampling.temp f32 = 1.000000
  32. llama_model_loader: - kv 5: general.name str = Qwen3.5-35B-A3B
  33. llama_model_loader: - kv 6: general.basename str = Qwen3.5-35B-A3B
  34. llama_model_loader: - kv 7: general.quantized_by str = Unsloth
  35. llama_model_loader: - kv 8: general.size_label str = 35B-A3B
  36. llama_model_loader: - kv 9: general.license str = apache-2.0
  37. llama_model_loader: - kv 10: general.license.link str = https://huggingface.co/Qwen/Qwen3.5-3...
  38. llama_model_loader: - kv 11: general.repo_url str = https://huggingface.co/unsloth
  39. llama_model_loader: - kv 12: general.base_model.count u32 = 1
  40. llama_model_loader: - kv 13: general.base_model.0.name str = Qwen3.5 35B A3B
  41. llama_model_loader: - kv 14: general.base_model.0.organization str = Qwen
  42. llama_model_loader: - kv 15: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen3.5-3...
  43. llama_model_loader: - kv 16: general.tags arr[str,2] = ["unsloth", "image-text-to-text"]
  44. llama_model_loader: - kv 17: qwen35moe.block_count u32 = 40
  45. llama_model_loader: - kv 18: qwen35moe.context_length u32 = 262144
  46. llama_model_loader: - kv 19: qwen35moe.embedding_length u32 = 2048
  47. llama_model_loader: - kv 20: qwen35moe.attention.head_count u32 = 16
  48. llama_model_loader: - kv 21: qwen35moe.attention.head_count_kv u32 = 2
  49. llama_model_loader: - kv 22: qwen35moe.rope.dimension_sections arr[i32,4] = [11, 11, 10, 0]
  50. llama_model_loader: - kv 23: qwen35moe.rope.freq_base f32 = 10000000.000000
  51. llama_model_loader: - kv 24: qwen35moe.attention.layer_norm_rms_epsilon f32 = 0.000001
  52. llama_model_loader: - kv 25: qwen35moe.expert_count u32 = 256
  53. llama_model_loader: - kv 26: qwen35moe.expert_used_count u32 = 8
  54. llama_model_loader: - kv 27: qwen35moe.attention.key_length u32 = 256
  55. llama_model_loader: - kv 28: qwen35moe.attention.value_length u32 = 256
  56. llama_model_loader: - kv 29: qwen35moe.expert_feed_forward_length u32 = 512
  57. llama_model_loader: - kv 30: qwen35moe.expert_shared_feed_forward_length u32 = 512
  58. llama_model_loader: - kv 31: qwen35moe.ssm.conv_kernel u32 = 4
  59. llama_model_loader: - kv 32: qwen35moe.ssm.state_size u32 = 128
  60. llama_model_loader: - kv 33: qwen35moe.ssm.group_count u32 = 16
  61. llama_model_loader: - kv 34: qwen35moe.ssm.time_step_rank u32 = 32
  62. llama_model_loader: - kv 35: qwen35moe.ssm.inner_size u32 = 4096
  63. llama_model_loader: - kv 36: qwen35moe.full_attention_interval u32 = 4
  64. llama_model_loader: - kv 37: qwen35moe.rope.dimension_count u32 = 64
  65. llama_model_loader: - kv 38: tokenizer.ggml.model str = gpt2
  66. llama_model_loader: - kv 39: tokenizer.ggml.pre str = qwen35
  67. llama_model_loader: - kv 40: tokenizer.ggml.tokens arr[str,248320] = ["!", "\"", "#", "$", "%", "&", "'", ...
  68. llama_model_loader: - kv 41: tokenizer.ggml.token_type arr[i32,248320] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
  69. llama_model_loader: - kv 42: tokenizer.ggml.merges arr[str,247587] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
  70. llama_model_loader: - kv 43: tokenizer.ggml.eos_token_id u32 = 248046
  71. llama_model_loader: - kv 44: tokenizer.ggml.padding_token_id u32 = 248055
  72. llama_model_loader: - kv 45: tokenizer.chat_template str = {%- set image_count = namespace(value...
  73. llama_model_loader: - kv 46: general.quantization_version u32 = 2
  74. llama_model_loader: - kv 47: general.file_type u32 = 15
  75. llama_model_loader: - kv 48: quantize.imatrix.file str = Qwen3.5-35B-A3B-GGUF/imatrix_unsloth....
  76. llama_model_loader: - kv 49: quantize.imatrix.dataset str = unsloth_calibration_Qwen3.5-35B-A3B.txt
  77. llama_model_loader: - kv 50: quantize.imatrix.entries_count u32 = 510
  78. llama_model_loader: - kv 51: quantize.imatrix.chunks_count u32 = 76
  79. llama_model_loader: - type f32: 301 tensors
  80. llama_model_loader: - type q8_0: 312 tensors
  81. llama_model_loader: - type q4_K: 78 tensors
  82. llama_model_loader: - type q5_K: 41 tensors
  83. llama_model_loader: - type q6_K: 1 tensors
  84. print_info: file format = GGUF V3 (latest)
  85. print_info: file type = Q4_K - Medium
  86. print_info: file size = 20.70 GiB (5.13 BPW)
  87. load: 0 unused tokens
  88. load: printing all EOG tokens:
  89. load: - 248044 ('<|endoftext|>')
  90. load: - 248046 ('<|im_end|>')
  91. load: - 248063 ('<|fim_pad|>')
  92. load: - 248064 ('<|repo_name|>')
  93. load: - 248065 ('<|file_sep|>')
  94. load: special tokens cache size = 33
  95. load: token to piece cache size = 1.7581 MB
  96. print_info: arch = qwen35moe
  97. print_info: vocab_only = 0
  98. print_info: no_alloc = 0
  99. print_info: n_ctx_train = 262144
  100. print_info: n_embd = 2048
  101. print_info: n_embd_inp = 2048
  102. print_info: n_layer = 40
  103. print_info: n_head = 16
  104. print_info: n_head_kv = 2
  105. print_info: n_rot = 64
  106. print_info: n_swa = 0
  107. print_info: is_swa_any = 0
  108. print_info: n_embd_head_k = 256
  109. print_info: n_embd_head_v = 256
  110. print_info: n_gqa = 8
  111. print_info: n_embd_k_gqa = 512
  112. print_info: n_embd_v_gqa = 512
  113. print_info: f_norm_eps = 0.0e+00
  114. print_info: f_norm_rms_eps = 1.0e-06
  115. print_info: f_clamp_kqv = 0.0e+00
  116. print_info: f_max_alibi_bias = 0.0e+00
  117. print_info: f_logit_scale = 0.0e+00
  118. print_info: f_attn_scale = 0.0e+00
  119. print_info: n_ff = 0
  120. print_info: n_expert = 256
  121. print_info: n_expert_used = 8
  122. print_info: n_expert_groups = 0
  123. print_info: n_group_used = 0
  124. print_info: causal attn = 1
  125. print_info: pooling type = 0
  126. print_info: rope type = 40
  127. print_info: rope scaling = linear
  128. print_info: freq_base_train = 10000000.0
  129. print_info: freq_scale_train = 1
  130. print_info: n_ctx_orig_yarn = 262144
  131. print_info: rope_yarn_log_mul = 0.0000
  132. print_info: rope_finetuned = unknown
  133. print_info: mrope sections = [11, 11, 10, 0]
  134. print_info: ssm_d_conv = 4
  135. print_info: ssm_d_inner = 4096
  136. print_info: ssm_d_state = 128
  137. print_info: ssm_dt_rank = 32
  138. print_info: ssm_n_group = 16
  139. print_info: ssm_dt_b_c_rms = 0
  140. print_info: model type = 35B.A3B
  141. print_info: model params = 34.66 B
  142. print_info: general.name = Qwen3.5-35B-A3B
  143. print_info: vocab type = BPE
  144. print_info: n_vocab = 248320
  145. print_info: n_merges = 247587
  146. print_info: BOS token = 11 ','
  147. print_info: EOS token = 248046 '<|im_end|>'
  148. print_info: EOT token = 248046 '<|im_end|>'
  149. print_info: PAD token = 248055 '<|vision_pad|>'
  150. print_info: LF token = 198 'Ċ'
  151. print_info: FIM PRE token = 248060 '<|fim_prefix|>'
  152. print_info: FIM SUF token = 248062 '<|fim_suffix|>'
  153. print_info: FIM MID token = 248061 '<|fim_middle|>'
  154. print_info: FIM PAD token = 248063 '<|fim_pad|>'
  155. print_info: FIM REP token = 248064 '<|repo_name|>'
  156. print_info: FIM SEP token = 248065 '<|file_sep|>'
  157. print_info: EOG token = 248044 '<|endoftext|>'
  158. print_info: EOG token = 248046 '<|im_end|>'
  159. print_info: EOG token = 248063 '<|fim_pad|>'
  160. print_info: EOG token = 248064 '<|repo_name|>'
  161. print_info: EOG token = 248065 '<|file_sep|>'
  162. print_info: max token length = 256
  163. load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
  164. load_tensors: offloading output layer to GPU
  165. load_tensors: offloading 39 repeating layers to GPU
  166. load_tensors: offloaded 41/41 layers to GPU
  167. load_tensors: CPU_Mapped model buffer size = 515.31 MiB
  168. load_tensors: CUDA0 model buffer size = 20685.78 MiB
  169. ..................................................................................................
  170. common_init_result: added <|endoftext|> logit bias = -inf
  171. common_init_result: added <|im_end|> logit bias = -inf
  172. common_init_result: added <|fim_pad|> logit bias = -inf
  173. common_init_result: added <|repo_name|> logit bias = -inf
  174. common_init_result: added <|file_sep|> logit bias = -inf
  175. llama_context: constructing llama_context
  176. llama_context: n_seq_max = 1
  177. llama_context: n_ctx = 16384
  178. llama_context: n_ctx_seq = 16384
  179. llama_context: n_batch = 2048
  180. llama_context: n_ubatch = 512
  181. llama_context: causal_attn = 1
  182. llama_context: flash_attn = auto
  183. llama_context: kv_unified = false
  184. llama_context: freq_base = 10000000.0
  185. llama_context: freq_scale = 1
  186. llama_context: n_ctx_seq (16384) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
  187. llama_context: CUDA_Host output buffer size = 0.95 MiB
  188. llama_kv_cache: CUDA0 KV buffer size = 320.00 MiB
  189. llama_kv_cache: size = 320.00 MiB ( 16384 cells, 10 layers, 1/1 seqs), K (f16): 160.00 MiB, V (f16): 160.00 MiB
  190. llama_memory_recurrent: CUDA0 RS buffer size = 62.81 MiB
  191. llama_memory_recurrent: size = 62.81 MiB ( 1 cells, 40 layers, 1 seqs), R (f32): 2.81 MiB, S (f32): 60.00 MiB
  192. sched_reserve: reserving ...
  193. sched_reserve: Flash Attention was auto, set to enabled
  194. sched_reserve: CUDA0 compute buffer size = 493.00 MiB
  195. sched_reserve: CUDA_Host compute buffer size = 40.02 MiB
  196. sched_reserve: graph nodes = 9399 (with bs=512), 3789 (with bs=1)
  197. sched_reserve: graph splits = 2
  198. sched_reserve: reserve took 73.29 ms, sched copies = 1
  199. common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
  200. clip_model_loader: model name: Qwen3.5-35B-A3B
  201. clip_model_loader: description:
  202. clip_model_loader: GGUF version: 3
  203. clip_model_loader: alignment: 32
  204. clip_model_loader: n_tensors: 334
  205. clip_model_loader: n_kv: 35
  206.  
  207. clip_model_loader: has vision encoder
  208. clip_ctx: CLIP using CUDA0 backend
  209. load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
  210. load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
  211. load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842
  212.  
  213. load_hparams: projector: qwen3vl_merger
  214. load_hparams: n_embd: 1152
  215. load_hparams: n_head: 16
  216. load_hparams: n_ff: 4304
  217. load_hparams: n_layer: 27
  218. load_hparams: ffn_op: gelu
  219. load_hparams: projection_dim: 2048
  220.  
  221. --- vision hparams ---
  222. load_hparams: image_size: 768
  223. load_hparams: patch_size: 16
  224. load_hparams: has_llava_proj: 0
  225. load_hparams: minicpmv_version: 0
  226. load_hparams: n_merge: 2
  227. load_hparams: n_wa_pattern: 0
  228. load_hparams: image_min_pixels: 8192
  229. load_hparams: image_max_pixels: 4194304
  230.  
  231. load_hparams: model size: 860.98 MiB
  232. load_hparams: metadata size: 0.12 MiB
  233. warmup: warmup with image size = 1472 x 1472
  234. alloc_compute_meta: CUDA0 compute buffer size = 248.10 MiB
  235. alloc_compute_meta: CPU compute buffer size = 24.93 MiB
  236. alloc_compute_meta: graph splits = 1, nodes = 823
  237. warmup: flash attention is enabled
  238. srv load_model: loaded multimodal model, '/home/user/.cache/llama.cpp/unsloth_Qwen3.5-35B-A3B-GGUF_mmproj-BF16.gguf'
  239. srv load_model: initializing slots, n_slots = 1
  240. common_speculative_is_compat: the target context does not support partial sequence removal
  241. srv load_model: speculative decoding not supported by this context
  242. slot load_model: id 0 | task -1 | new slot, n_ctx = 16384
  243. srv load_model: prompt cache is enabled, size limit: 8192 MiB
  244. srv load_model: use `--cache-ram 0` to disable the prompt cache
  245. srv load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
  246. init: chat template, example_format: '<|im_start|>system
  247. You are a helpful assistant<|im_end|>
  248. <|im_start|>user
  249. Hello<|im_end|>
  250. <|im_start|>assistant
  251. Hi there<|im_end|>
  252. <|im_start|>user
  253. How are you?<|im_end|>
  254. <|im_start|>assistant
  255. <think>
  256. '
  257. srv init: init: chat template, thinking = 1
  258. main: model loaded
  259. main: server is listening on http://127.0.0.1:8080
  260. main: starting the main loop...
  261. srv update_slots: all slots are idle
  262. srv log_server_r: done request: GET / 127.0.0.1 200
  263.  
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!