llama.cpp adds a CPU-share dial to its MoE expert cache, and a script to tune both
llama.cpp merged a flag that runs missed MoE experts on the CPU and a script that tunes both cache knobs. The author's own numbers show why the split matters.
Vlad Makarovreviewed and published
llama.cpp merged PR #30966 on 7 November. With --moe-cache-cpu-share set, the CPU computes a share of the MoE experts the GPU expert cache misses, work that would otherwise be an upload from system memory. The same pull request, by contributor am17an, adds scripts/moe-cache-tune.py, which measures a machine and prints the two cache arguments it should use. The diff is 1,169 lines added and 183 removed across 18 files. This is not a new engine or a release but the tuning layer for the cache that landed in October, and its evidence is one workstation running one model.
What the dial actually changes
The existing cache keeps the most-used experts in device memory and runs their matrix multiplications on the GPU. The new parameter does not move that boundary wholesale: hits still run on the device, and misses are still uploaded into the cache. What the share changes is the fate of misses once the cache is full. The documentation added in the same PR calls it the "share of the cache misses computed on the CPU from the host experts instead of uploaded", defaulting to 0, the old behaviour. The reasoning is bandwidth arithmetic: if the CPU can read an expert out of host memory faster than the GPU can pull the same weights across PCIe, handing it the work beats copying it. The docs also name the exception that decides most outcomes. Pinned experts, as with -lm mlock or -lm dio, upload at the full speed of the link, and "the CPU share often does not help then". The default mmap'd weights are not pinned, which is where the share earns its keep.
What the author measured
The PR body carries one example run on Qwen3.8-Flash-Next at IQ4_NL, on a machine reporting 64,800 MiB of experts in host memory and 92,658 MiB free on CUDA0, with up to 24,000 MiB allowed for the cache. Six calibration runs at two cache sizes and three shares produced this:
| Cache MiB | Share 0 | Share 0.5 | Share 0.95 |
|---|---|---|---|
| 6,464 | 24.2 t/s (73.3%) | 29.6 (63.4%) | 47.6 (61.5%) |
| 19,392 | 65.5 (93.5%) | 66.4 (91.9%) | 54.1 (71.6%) |
Bracketed numbers are cache hit rates. Across the top row the flag looks like a large win: at a 6,464 MiB cache, share 0.95 raises decode from 24.2 to 47.6 tokens per second, 97% better, even as the hit rate falls. Across the bottom row the same move loses 17%, from 65.5 to 54.1. One flag, opposite results, decided by how much VRAM the cache was given.
Why the two dials pull against each other
At the small cache most tokens miss and the upload path is the bottleneck, so letting the CPU serve some misses removes work from the slowest stage. At the large cache 93.5% of lookups already hit; routing the rest to the CPU trades away a nearly complete cache, and the reported hit rate falls to 71.6%. The author's predicted table, which replays the recorded routing at other settings, makes the crossing point concrete: 12,928 MiB at share 0.95 predicts 65.9 tokens per second against 66.0 for 19,392 MiB at share 0. That is 6,464 MiB of VRAM for a rounding difference, and not a trade anyone can guess from a spec sheet. The script's recommendation rule follows: the lowest share within 5% of the fastest, which here is 24,000 MiB at share 0, a predicted 83.0 tokens per second.
The script is the real deliverable
am17an wrote a tuner rather than a recommended default. It runs llama-server six times, at two cache sizes and shares of 0, 0.5 and 0.95, generates 4,096 tokens on the first run and 1,024 on the others, and reads the model once so disk I/O does not contaminate the timings. From those runs it fits two machine constants, the upload rate and the CPU expert read rate, then replays the routing trace through a copy of the cache logic to predict other sizes and shares. The fit is close: at 6,464 MiB and share 0.95 the prediction was 48.1 tokens per second against 47.6 measured. The machine profile it prints explains the result better than any single rate: 9.6 GiB/s uploads, 48.7 GiB/s for CPU-resident experts, 2.99 ms per token in the CPU part, an overlap term of -19.9 ms/GiB, and 6.82 ms per token for everything else. Invocation is a prefix on the command you already use:
python3 scripts/moe-cache-tune.py -- build/bin/llama-server -m model.gguf -cmoe -t 8 -c 32768
Runs are reused while the server command stays the same, so trying another --max-mib is instant.
What the numbers do not cover
The example is one model at one quant on one machine, and that machine has 92,658 MiB free on its first CUDA device. That is not the 10 or 12 GB card the low-VRAM crowd runs, which is where the change surfaced, in an r/LocalLLaMA thread describing it as a speedup for MoE offloading. Nothing in the pull request measures that case. The predicted columns are a replay of recorded routing through a software model of the cache, not fresh measurements, and the only check on it in the body is the agreement between one predicted cell and one measured one. No third party has reproduced the result, and no release tag accompanies the merge. The contributor also disclosed AI assistance in the required checkbox, "YES, paired with Claude", a disclosure about process rather than correctness.
Where it sits
This is the second step in a line of work rather than a standalone feature. PR #29887 added the cache itself in October with -cmoe and --moe-cache-mib, and left the size as a number users guessed at. A competing design in issue #29949 proposes a GPU-resident LRU with eviction decided on the device and reports 9.7 to 37-50 tokens per second on a 24 GB Radeon RX 7900 XTX, 3.5x to 4x depending on workload, on its author's own build; the issue is still open. What #30966 adds is a second dial, a script that sets both, and documentation the earlier patch arguably lacked: a 26-line docs/moe-cache.md plus one-line mentions in the CLI, completion and server READMEs.
What to do with it
If you already run -cmoe, put the tuner in front of your own server command, keep your own thread and context settings, and let it report the hit rates rather than assuming a gain. Check whether your experts are mmap'd or pinned first, because pinned weights make the CPU share close to useless. Then compare its recommendation against a manual run before keeping it: the point of the script is that the best pair of numbers here would be wrong on your box, and the author's own table shows a 97% gain and a 17% loss separated only by the cache size next to it.
