llama.cpp Merges Qwen3.8-Flash-Next Support — GGUF Is Coming, Hardware Not Included

llama.cpp merged support for Alibaba's Qwen3.8-Flash-Next on August 27, the day after the model dropped — making the exotic 125B mixture-of-experts runnable in the default local-inference engine. Pull request #27742 landed in ggml-org/llama.cpp, and r/LocalLLaMA noticed: "finally I can download the GGUF," one commenter wrote on a thread that drew roughly 330 points.

What Happened

The PR — "model: add Qwen3.8-Flash-Next (qwen4exp)" — comes from Daniel Han, the founder of Unsloth, which shipped support within hours of release. This was no formality. Qwen3.8-Flash-Next combines Gated DeltaNet recurrence with Qwen Sparse Attention, hyper-connections, and per-layer embeddings backed by a 97.7 GiB n-gram hash table offloaded to host memory. Support took a new converter, a sparse-attention graph, a third cache in the hybrid memory system, vision support, and three quantizer fixes. Notably, ggml/ itself is untouched: no new ops were needed.

Correctness checks against the reference are unusually thorough — wikitext-2 perplexity within noise (4.0068 vs 4.0126), 98 percent top-1 agreement, QSA bit-identical to dense attention below its budget. One quantizer fix matters beyond this model: a work-buffer sizing bug that could allocate 150 GB per process on the 51.2-billion-element embedding tensor. Qwen's README now lists llama.cpp under Local Use — GGUF for text and vision.

The Specs

• 125B backbone + 51B N-gram embedding table + 4B MTP module = 180B total, 6B active per token
• 512 experts, top-10 routing plus a gated shared expert
• Gated DeltaNet in 3 of every 4 layers, QSA in the fourth
• 262,144-token native context
• FP8 checkpoint: roughly 173 GiB

Why This Matters

llama.cpp is where most people actually run open-weights models: GGUF quantization, CPU/Apple Silicon/NVIDIA support, one command. That this architecture works there at all — recurrence, sparse attention, host-offloaded embeddings — is a real milestone, and the turnaround was fast: feature request #27741 went in right before the merge, roughly two days after release — the first mainstream local support after Unsloth, vLLM, and SGLang.

Keep the enthusiasm measured, though. "Local" here means a big rig, not a laptop. The FP8 checkpoint is 173 GiB; early PR-thread tests run heavily quantized builds on 128 GB unified-memory hardware. A 6B-active MoE with recurrent layers also serves unusually: the PR notes quantized models are not bit-identical across the sparse-attention boundary, so expect quantization-quality reports to diverge from the dense reference. GGUF support is real progress; "runs anywhere" is a stretch.

What's Next

Expect GGUF uploads to hit Hugging Face over the coming days — Qwen's README points there — plus community quantization tests. For most users this is the first realistic way to touch Qwen3.8-Flash-Next at all; our launch coverage has the architecture details.