Betelgeuse runs a 68GB Qwen build on a 12GB card, and the SSD sets the pace
Betelgeuse, a llama.cpp fork, decodes a 68GB Qwen 3.8 Flash Next build at 22 tok/s on a 12GB RTX 3060 by streaming experts off an NVMe drive, bit-exact.
Vlad Makarovreviewed and published
Betelgeuse is a fork of llama.cpp that decodes a 68GB quantised build of Qwen 3.8 Flash Next at 22.03 tok/s on an RTX 3060 with 12GB of VRAM and 16GB of DDR4. The author posted the result on r/LocalLLaMA on 31 October, next to a public repository and a model on Hugging Face. Stock llama.cpp on the same machine manages 1.47 tok/s.
The gap is not a faster kernel or a smaller model. All ten routed experts per layer, across all 48 layers, are still computed at full precision. What changed is where the bytes live between tokens, which makes this a memory-planning result that happens to be fast.
The bottleneck was always bytes per token
A decoder that reads its weights from storage is a bandwidth problem in a compute problem's costume. Stock llama.cpp maps the GGUF and lets the page cache decide what stays resident: roughly 1,500 major page faults per token, and 208 to 312 MiB off the drive for every token emitted. Betelgeuse's own counter reads 12.43 MiB per token, a factor of seventeen to twenty-five, which accounts for most of the speedup.
Physical reads are the fairest number here: a block-device counter can be checked without trusting anyone's stopwatch. At 12.43 MiB the transfer is about 13MB, a little over six milliseconds at the drive's rated 2.1 GB/s, against the 45 milliseconds a token takes at 22.03 tok/s. The pipe is not saturated; latency is. Expert blobs are 4KB-aligned single reads scattered through a 35.3GB file, and a miss has to ring a doorbell, wake the host and wait. Six milliseconds of reading inside a forty-five millisecond token is the real engineering surface, which is why throughput tracks hit rate rather than bandwidth.
What the fork actually does
About 5,000 expert slots sit in VRAM, allocated unevenly: 240 per layer on layers 0 to 4, 92 through the middle, 60 on the tail. The 49 dense layers, the GDN and QSA blocks and the LM head stay resident in roughly 3.2GB. A custom CUDA kernel resolves a hit on-device in about two microseconds; only a miss crosses to the host, through a mapped doorbell. The author says the stock ggml_backend_sched path spent about 360 milliseconds per token in cudaStreamSynchronize on the same decisions, longer than an entire token now takes.
The decode pass is one CUDA Graph replay. A RAM victim cache and an io_uring pager keep the tiers exclusive: an expert is in VRAM, RAM or disk, never two at once, so system memory holds no second copy of what the card has.
Three ways to run the same file on the same box
| Setup | Decode | NVMe reads |
|---|---|---|
| Stock llama.cpp, mmap | 1.47 cold / 1.63 warm tok/s | 208-312 MiB/token |
Strata, forced --low-ram mmap | 8.11 tok/s | not reported |
| Betelgeuse | 16.80 cold / 22.03 warm tok/s | 12.43 MiB/token |
Strata is the fairest baseline, because it attempts the same trick. With 48GB to 64GB of RAM it can mlock 24GB or more of experts and reports above 40 tok/s. On 16GB it refuses to load unless forced, then lands at 8.11 tok/s, bound by cold SSD reads. Betelgeuse reaches 22.03 warm under that constraint, about 2.7 times Strata's forced mode, and both sit below what enough RAM buys. Streaming rescues the machine; RAM is still what makes it fast.
The prototype he threw away
Two months earlier the same author posted a prototype that predicted which experts a token would route to, then withdrew it. His explanation names the trick that many speed claims in this genre quietly rely on: "to get the big numbers I was seeing back then, I was basically cheating with gate pruning (dropping cold experts when their router weights were low)". Pruning degraded output, and he says so: "I didn't want to ship something that lobotomizes the model just to look fast in a screenshot, so I scrapped it."
The admission matters more than the benchmark, because it sets up a checkable claim. An IQ2_XS build has little precision left to spend; dropping experts on top compounds one approximation with another. The evidence against it is narrow but direct: 28,080 router records and 576 generated tokens against golden stock output, maximum difference 0.000e+00.
3.2GB of dense weights is what buys the slot pool
Qwen 3.8 Flash Next carries a 125B-parameter backbone plus 51B of N-gram embeddings in the PLE shard, activating a small slice per token, so the headline is not that a big model ran on a small card. It is that 28GB of VRAM and RAM was enough, because only the dense part must be resident, and it is small. Three point two gigabytes leaves most of a 12GB card for expert slots, and the slot count sets the hit rate, which sets the bytes per token.
The layout is a 3.2GB Dense.gguf, a 35.3GB experts.bin of 4KB-aligned single-read blobs, and a 26.8GB PLE.gguf n-gram shard, instead of roughly 35GB of dead expert tensors duplicated inside Shard 1 and again in experts.bin. Those files sum to 65.3GB, the author calls the build 68GB, and Hugging Face reports 71.8GB stored: close enough to be one artifact, far enough apart that none should be quoted as another.
One quant, one box, one author
The scope is narrower than the thread suggests. One model, one quantisation, one deliberately handicapped machine: he says he pulled his extra RAM sticks to test what a regular 16GB desktop gets, ran inside a MemoryMax=12G cgroup on CachyOS, and measured against single-channel DDR4 at about 19 GB/s. Every number is self-reported, nobody else has reproduced it, and the 48 tok/s figure is a burst where all ten routed experts were already resident, not a sustained rate.
The upstream pull request, which adds --moe-direct-io and the slot pool, was opened on 31 October and remains open and unmerged, so today this is a fork you build rather than a flag you pass. The repository is MIT-licensed with 70 stars and one fork; the model card reports 521 downloads, 4 likes and an "other" licence. The parity test is also short: 576 tokens is strong evidence that no experts were dropped and weak evidence about a long generation, where a drifting hit rate would show up as declining speed rather than wrong text.
He says the same architecture is coming to DeepSeek V4.1 Flash, with early testing around 25 tok/s on the same hardware class. If that holds, the claim stops being about one Qwen build and becomes a statement about MoE inference on consumer hardware: the ceiling is storage bandwidth and how well the next token's expert set can be guessed, and the dense fraction decides whether the guess is affordable. Betelgeuse is a credible first data point, published with its numbers in the open, and one replication short of more.
