Qwen3.8-27B on an RTX 3090: Inference Options and Benchmark Results

Oct 24, 2026 · @Someone
Summary

HyperQwen (a patched vLLM in Docker) is the fastest way to run Qwen3.8-27B on an RTX 3090: 133 tokens/s single-stream with a 94 ms time to first token, 3.2x the 41 t/s of the stock llama.cpp router. Among llama.cpp builds, the llamAmpere fork with MTP speculative decoding at depth 2 and its turbo4 KV cache reaches 73 t/s (1.76x) and keeps the router, the GGUF models and the 64k context intact. Compiling stock llama.cpp specifically for sm_86 changes nothing, and on stock llama.cpp every speculative-decoding preset (MTP, DFlash2) is slower than plain decoding.

The work shipped as NixOS specialisations (llm-fa86, llm-llamampere), a benchmark tool (llm-bench), a corrected base preset set, and a Docker setup for HyperQwen outside Nix. Two decisions remain open: whether to make the llamAmpere configuration the router default, and whether HyperQwen earns a permanent place as an alternative engine.
Setup and method

All numbers come from one machine, one model and one protocol, measured on 2026-10-24.

Item


Value

Host


NixOS unstable (nixpkgs 02c087f), kernel 6.18, 32 GB RAM

GPU


NVIDIA GeForce RTX 3090, 24 GB, driver 595.104.02, compute capability 8.6

Model (llama.cpp)


unsloth/Qwen3.8-27B-GGUF, UD-Q4_K_XL (17.6 GB), text only, context 65536, --fit on

Model (HyperQwen)


Qwen3.8-27B W4A16 AutoRound with requantized heads, prepared by the HyperQwen prepare step

Serving stack


services.llama-cpp router (--models-preset), port 48123, models load on demand and unload after 300 s idle

Benchmark


llm-bench: one fixed coding prompt, 512 output tokens, 3 runs, temperature 1.0, seed 42, model unloaded after each alias

Metrics


decode t/s from the server's timings (median of 3); TTFT = wall clock to the first streamed token with the model loaded; cold load = warm-up request after unload; draft acceptance = accepted / drafted tokens

Before each load


VRAM in use was 1.6 to 1.8 GB (desktop, Sunshine, ComfyUI); a run with more than ~2 GB taken is not comparable

HyperQwen exposes no server-side timings, so a Python script with the same prompt and parameters measured it through its OpenAI-compatible streaming API: decode t/s = tokens after the first divided by the time from the first to the last chunk.
Options tested

Four options were tested. Only engines that need different compiled CUDA kernels became NixOS specialisations; the vLLM-based engine runs in Docker outside Nix.

Option


What changes


How it is activated

Base config fix


nixpkgs llama.cpp 0.6.0 as shipped. The three Qwen3.8 presets moved from KV q8_0/q4_0 to q8_0/q8_0, because the stock build has no FlashAttention kernel for the mixed combination and fell back to an f16 conversion on every step. New control preset qwen3.8-27b-base with no speculative decoding.


sudo nixos-rebuild switch --flake .#

llm-fa86


Same llama.cpp, rebuilt with CMAKE_CUDA_ARCHITECTURES=86 and GGML_CUDA_FA_ALL_QUANTS=ON (every KV quant combination gets a kernel, sm_86 only). Adds preset qwen3.8-27b-kv84 (MTP depth 2, KV q8_0/q4_0) to check whether the missing kernel was the whole problem.


sudo nixos-rebuild switch --specialisation llm-fa86 --flake .#

llm-llamampere


llamAmpere v0.5.5, a llama.cpp fork tuned for Ampere: verify matvec kernels, a q4_K/q5_K/q6_K routing table, compressed KV types turbo2 to turbo6. Built with the nixpkgs recipe and the same two CUDA flags. Adds presets ampere-none (true control, spec-type = none), ampere-mtp4, ampere-turbo4 (depth 4, KV turbo4) and ampere-turbo2 (depth 2, KV turbo4).


sudo nixos-rebuild switch --specialisation llm-llamampere --flake .#

HyperQwen


HyperQwen: vLLM 0.31 patched with int8 Marlin and int8 flash-attention kernels for sm_86, prebuilt image ghcr.io/syv-ai/hyperqwen:latest, its own W4A16 model. Tested in two modes: SPEC=dflash2 (the .env.example default) and SPEC=mtp (the launcher default).


docker compose --profile single up -d; API on port 18020

Switching between the base and a specialisation restarts only llama-cpp.service; no reboot is needed and the model cache in /var/cache/llama-cpp is shared. The specialisations add their presets through a new services.llama-cpp.presets option that the base module renders to the router's INI file.
Results
HyperQwen decodes 133 t/s, 3.2x the stock llama.cpp baseQwen3.8-27B on an RTX 3090, median of 3 runs, 512 tokens; dashed line = stock baseHyperQwen dflash2 (ctx 40k)133.4HyperQwen mtp (ctx 40k)123.2llamAmpere turbo2: MTP d2, KV turbo472.9llamAmpere: MTP d2, KV q842.8llamAmpere turbo4: MTP d4, KV turbo442.5llamAmpere: no speculation41.6stock llama.cpp: no speculation41.3llm-fa86: no speculation41.2llamAmpere: auto (adaptive MTP)26.4llamAmpere: MTP d4, KV q822.7llm-fa86 kv84: MTP d2, KV q8/q422.0stock: MTP d2, KV q819.1stock Turbo-Coder: MTP d218.8llm-fa86: MTP d2, KV q818.7stock: DFlash2 d413.4llm-fa86: DFlash2 d413.0stock base 41.3 t/s

Everything above the dashed line beats plain decoding on the stock router; on stock llama.cpp no speculative preset does. The table carries the rest of each measurement. Depth = speculative draft tokens per step (spec-draft-n-max).

Configuration


Engine


Speculation


KV k/v


Draft acc


TTFT ms


Cold load s


Context

HyperQwen dflash2


vLLM 0.31 patched


DFlash2 drafter


int8


n/a


94


260 (compile)


40k

HyperQwen mtp


vLLM 0.31 patched


MTP


int8


n/a


110


270 (compile)


40k

llamAmpere turbo2


llamAmpere 0.5.5


MTP depth 2


turbo4/turbo4


0.69


226


8.3


64k

llamAmpere MTP d2


llamAmpere 0.5.5


MTP depth 2


q8_0/q8_0


0.60


282


8.9


64k

llamAmpere turbo4


llamAmpere 0.5.5


MTP depth 4


turbo4/turbo4


0.43


276


7.0


64k

llamAmpere none


llamAmpere 0.5.5


none


q8_0/q8_0


-


239


12.0


64k

stock base


nixpkgs llama.cpp 0.6.0


none


q8_0/q8_0


-


266


6.5


64k

llm-fa86 base


llama.cpp 0.6.0 sm86


none


q8_0/q8_0


-


264


10.6


64k

llamAmpere auto


llamAmpere 0.5.5


fork default: adaptive MTP


q8_0/q8_0


0.48


424


9.4


64k

llamAmpere MTP d4


llamAmpere 0.5.5


MTP depth 4


q8_0/q8_0


0.43


385


8.4


64k

llm-fa86 kv84


llama.cpp 0.6.0 sm86


MTP depth 2


q8_0/q4_0


0.58


497


9.3


64k

stock MTP


nixpkgs llama.cpp 0.6.0


MTP depth 2


q8_0/q8_0


0.50


466


21.5


64k

stock Turbo-Coder


nixpkgs llama.cpp 0.6.0


MTP depth 2 (DavidAU finetune)


q8_0/q8_0


0.65


543


33.4


64k

llm-fa86 MTP


llama.cpp 0.6.0 sm86


MTP depth 2


q8_0/q8_0


0.50


448


20.9


64k

stock DFlash2


nixpkgs llama.cpp 0.6.0


DFlash2 depth 4


q8_0/q8_0


0.44


514


13.3


64k

llm-fa86 DFlash2


llama.cpp 0.6.0 sm86


DFlash2 depth 4


q8_0/q8_0


0.48


763


15.6


64k

llamAmpere DFlash2


llamAmpere 0.5.5


DFlash2


q8_0/q8_0


fails to load (dflash requires ctx_other)


-


-


-
Findings

    The missing FlashAttention kernel was not the whole story. The stock build logged no FlashAttention vector kernel compiled for K/V types q8_0-q4_0, converting K and V to f16 instead (slow) for every Qwen3.8 preset. Moving to q8_0/q8_0 removes the warning, yet every speculative preset on stock llama.cpp still loses to plain decoding: each MTP step costs more than twice a plain decode step, so a 50 to 65 percent draft acceptance cannot pay for it.

    Compiling for sm_86 alone changes nothing. llm-fa86 matches the stock build within noise on every preset. The kv84 preset (q8_0/q4_0 with the kernel present) reaches 22 t/s, better than the 19 t/s of q8_0/q8_0 MTP but still half the plain base.

    llamAmpere's plain decode equals upstream (41.6 vs 41.3 t/s). Its gain is entirely in the speculative path: MTP depth 2 goes from 18.7 to 42.8 t/s on the same q8_0 KV, and with its turbo4 KV cache to 72.9 t/s with 0.69 acceptance. Depth 4 is slower than depth 2 on every KV type, and the fork's automatic adaptive MTP (26.4 t/s) is worse than an explicit depth 2.

    HyperQwen is in a different class: 133 t/s and 94 ms TTFT with the DFlash2 drafter, 123 t/s with MTP, with run-to-run spread under 0.5 t/s. It is the only option that exceeds 2x the base.

    Speculation costs first-token latency on llama.cpp. Drafted presets add 200 to 300 ms of TTFT on stock llama.cpp; only the llamAmpere turbo presets bring TTFT below the plain base (226 vs 266 ms).

    Cold start matters with a 300 s idle unload. The first request after idle pays 6 to 9 s on the fork, 21 to 33 s for the stock MTP presets (the Turbo-Coder finetune is the slowest to load), and 4 to 5 minutes for HyperQwen, which also holds 21 to 23 GB of VRAM while it runs.

Caveats and method notes

    VRAM contention invalidates a run. An early pass read 15 t/s for the stock base because another process held VRAM when --fit sized the context, and layers went to the CPU without any warning in the log. llm-bench now prints VRAM in use before each load; the reported runs all started at 1.6 to 1.8 GB.

    HyperQwen ran at 40k context, not 64k. Its launcher pins a 5.2 GiB KV pool assuming an empty card and ran out of memory during warm-up with the 1.7 GB the desktop, Sunshine and ComfyUI hold. The test used GPU_UTIL=0.90, KV_MEM=3972844748 and MAX_LEN=40960. With the card free the stock 64k should fit; this was not verified.

    HyperQwen serves a different quantization. W4A16 AutoRound with requantized heads and its own drafters, not the UD-Q4_K_XL GGUF. Both are 4-bit, but output quality was not compared.

    turbo4 compresses the KV cache to about 4 bits. Its effect on answer quality at long context was not measured; a real-use check is needed before making it the default.

    Speculative decoding is not bit-exact with plain sampling by design; the benchmark measures speed only.

    Docker on this host uses CDI, not the legacy NVIDIA runtime. A local docker-compose.override.yml replaces the stock driver: nvidia reservation with driver: cdi and nvidia.com/gpu=all.

    The llamAmpere fork does not support draft-dflash, and its --spec-type defaults to auto, so a preset without spec-type is not a control there.

    Short prompt (41 tokens for llama.cpp, 83 for vLLM's template), so prompt-processing numbers were not compared.

Recommendations and open decisions

    For the router (keeps GGUF models, 64k context, multi-model on demand): adopt the llamAmpere build as the default services.llama-cpp.package and set the Qwen3.8-27B preset to MTP depth 2 with cache-type-k/v = turbo4. Expected: 73 t/s and 226 ms TTFT instead of 19 t/s and 466 ms today. Precondition: a quality check of turbo4 in real coding sessions.

    For the fastest single-user Qwen3.8-27B: run HyperQwen in single mode with SPEC=dflash2. It needs the whole GPU, so it is an alternative to the router, not a companion; a small wrapper that unloads the router's models (or stops llama-cpp.service) before docker compose up would make the switch safe. Verify 64k context with the card free before relying on it.

    Drop the DFlash2 preset from the stock router: it is the slowest option on every llama.cpp build and does not load on the fork.

    Optional fine sweep on llamAmpere: KV turbo3, turbo5, turbo6, depths 1 and 3, and draft-mtp-adaptive with turbo4. About six presets, one specialisation switch.

Sources: llamAmpere; HyperQwen.