Qwen3.8-Flash-Next — speed & llama.cpp config (HN reply draft)

Hardware: Framework Desktop, Strix Halo (Ryzen AI MAX+ 395), 128 GB unified RAM.

Stack: Nathanw strix-halo-llamacpp v0.7.6.1 Vulkan (llama-server), not stock llama.cpp. Weights: AtomicChat AD-4.27bpw-Q4_K_M-M64 (33 shards; ~54.5 GB transformer in fast memory, ~38.4 GB n-gram/PLE shard mmap'd from NVMe). Shared MTP sidecar: Unsloth mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf. Chat template: froggeric fixed Qwen template (not the broken GGUF default).

Speed (real traffic, LiteLLM on my agent stack, median prompt ~50–55k tokens, thinking on):
                                                        • Overall decode p50: ~30 tok/s
• 50k+ prompt (typical agent turn): ~26 tok/s
• Shorter turns (<8k): ~38–39 tok/s
• Cold prefill (llama-server prompt eval, 13k+ tok): ~450 tok/s p50
• TTFT: ~2 s p50 / ~9 s p90

Harness at 55k agentic: warm repeat decode ~42–47 tok/s; cold first-fill prefill ~471 tok/s (v0.7.6). Bench numbers with identical cached 55k prompts can hit 90+ tok/s — that's turn-2+ KV reuse, not first message after load.

Flags that matter for this model (different from normal Strix Halo recipes):
• Do NOT use --no-mmap (n-gram table must page from disk)
• MUST use -fit off (auto-fit mis-sizes qwen4exp)
• MUST use --load-mode mmap --no-host --no-repack --tensor-read-lazy on on Vulkan
• -ot per_layer_token_embd.weight=CPU --n-cpu-moe 0

Sampling (thinking on): temperature=1.0 top_p=0.95 top_k=20 min_p=0.0

Short version for HN: ~26 tok/s decode at ~55k context on Nathanw v0.7.6 Vulkan + AD-4.27 + mmap flags + MTP.

---

llama-server command:

~/Code/strix-halo-llamacpp-v0.7.6.1/vulkan/llama-server --host 0.0.0.0 --port 8080 \
  -m ~/Secondary/Models/Qwen3.8-Flash-Next/AD-4.27bpw/Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64/Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64-00001-of-00033.gguf \
  -md ~/Secondary/Models/Qwen3.8-Flash-Next/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \
  --mmproj ~/Secondary/Models/Qwen3.8-Flash-Next/mmproj-Qwen3.8-Flash-Next-F16.gguf \
  --image-min-tokens 1024 \
  -fa on -ngl 999 -b 8192 -ub 2048 -t 4 -fit off \
  --ctx-size 262144 \
  --cache-type-k q8_0 --cache-type-v q8_0 --cache-reuse 1 \
  --spec-type draft-mtp,ngram-mod --spec-draft-n-max 6 --spec-draft-p-min 0.75 \
  --jinja --chat-template-file ~/llama-swap/templates/chat_template.jinja \
  --reasoning-format deepseek \
  --reasoning-budget 8192 --reasoning-budget-message "Let me provide my answer now." \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 \
  --load-mode mmap --no-host --no-repack --tensor-read-lazy on \
  -ot per_layer_token_embd.weight=CPU --n-cpu-moe 0 --parallel 1 \
  --slot-save-path ~/llama-swap/slots