Gemma 4 at 3.81B: the 85% that was removed was the routing
A community researcher pruned Gemma 4 26B-A4B to 3.81B parameters. The arithmetic shows the 85% he cut was unused routing capacity, not work done per token.
Vlad Makarovreviewed and published
Everett Wilber, an independent developer posting as Accomplished_Ad_56, removed 85 percent of Gemma 4 26B-A4B's weights and shipped a 3.81B-parameter model that agrees with the original on 91.7 percent of 906 held-out screen-activity ticks. His write-up is candid about the failures; the thread filled with lobotomy jokes anyway. The number worth arguing about is not the percentage. Size fell by 85 percent, and the work done per token did not fall at all.
What the A4B suffix already told you
Google's model card lists 25.2B total parameters and 3.8B active, with 8 of 128 experts firing per token. The A4B suffix is not rounding on a big model. It describes a forward pass, in which most of the weights exist to be chosen between.
The published config allows a fuller count: 30 layers, hidden size 2816, 128 experts at intermediate size 704, and a 262,144-row embedding. Attention and the dense MLP come to 1.68B, the embedding to 0.74B, and pruning leaves both untouched.
| Weight group | 26B-A4B | At 8 experts per layer |
|---|---|---|
| Experts, 128 down to 8 per layer | 22.84B | 1.43B |
| Attention plus dense MLP | 1.68B | 1.68B |
| Token embedding, 262,144 rows | 0.74B | 0.74B |
| Total | 25.26B | 3.84B |
| Parameters used per token | 3.8B | 3.84B |
The re-derived total sits within 0.3 percent of Google's 25.2B, and the pruned count within one percent of the author's 3.81B. The compressed model is the same size as one forward pass of the original, and per-token computation did not shrink. What disappeared is the router's option to pick a different eight experts for the next token, and with it every combination the model could previously reach.
The rare experts were the load-bearing ones
The obvious method failed outright. Wilber first kept the experts that fire most often, and the model collapsed; experts that almost never activate turned out to be holding the prompt together. Selecting by ablation, removing an expert and measuring how far the hidden state moved, was the rule that worked. That is the most transferable idea in the project.
Reading and writing had to be split. Prefill kept collapsing even at sixteen experts per layer; decode pruned easily. The repair pinned each block's output at its global-attention layer to the original's, then trained each block on the original's clean input instead of the pruned model's damaged output. Block-local training beat end-to-end at the same budget, 0.2995 against 0.3312 on a hidden-state error that started at 0.6858.
The final run uses LoRA on the surviving experts at rank 128, a dense adapter at rank 64 and a router correction: about 369 million trainable parameters over roughly 4,000 steps on 3,016 prompts. Claude wrote the code and ran it on Wilber's laptop.
91.7% is agreement with a teacher, not accuracy
The headline compares the pruned model to the 26B on 906 ticks it never saw, with a 95 percent interval of 89.8 to 93.3. The unmodified 12B scores 90.0 on the same set. Wilber set his own bar at 98 and says he missed it.
The aggregate hides the error profile, which is what matters for a monitor running all day. The pruned model missed drift 18 times where the 12B missed it 3, and raised 57 false "not aligned" verdicts against the 12B's 88. Two models agreeing 91.7 and 90 percent are not interchangeable if one fails where the app becomes useless.
MMLU changed the picture. Agreement on held-out biology fell to 65 percent, and accuracy against the exam key dropped from 97 to 68, with 55 percent on neighbouring subjects, 52 mid-distance and 46 far out. His first test had produced identical top answers on all 70 prompts, a sample size where that should have read as a warning.
The admission in the thread is more useful than the blog. Asked about hand labels, Wilber said he had "a LOT of labeled data ... I haven't used yet", and had "decided to just focus on metrics that measure teacher reconstruction for the post". The 26B disagreed with his own labels on 15 of 67 ticks, so agreement was always the easier target.
"So let me get this straight, you took a generalist 26B-A4B, and kept just that A4B for one specific task?"
u/slvrsmth wrote that, and it is less a criticism than a restatement. u/WallaceGrommet named the thing nobody measured: the model "can still agree 92% of the time while having no idea when it's guessing". No calibration numbers appear anywhere in the post.
The published weights are worse than the post is
The classifier is not downloadable, because it was trained on Wilber's own screen data. Two other artifacts are. His en-ja translator strips 120 of 128 experts from every layer and ships as a 2,753,442,560-byte Q4_K_M file, the same size class as the classifier.
Its evaluation is the most honest thing in the release. Fed the original's translation one token at a time on WMT24++, the pruned model's top choice matches the original's 74.25 percent of the time, against 51.89 percent for the same cut with no healing, so the repair does real work. But when both translate freely, the outputs were identical on 2 of 150 rows, with a mean character similarity of 0.64. The card lists the symptoms: "2.5 mg" emerging as "25 mg", code translated as prose, specialist terms garbled into katakana. Swahili, Wilber adds, is markedly worse.
That is what general degradation looks like once somebody measures it.
The file size checks out, and explains the speed
Parameters times bits gives the file. At Q4_K_M's nominal 4.5 bits per weight, 3.81B parameters should occupy 2.14 GB. The published translator is 2.75 GB, or 5.78 bits per parameter. The gap is not an error: a K-quant does not hold every tensor at its headline rate, and the embedding is 19.2 percent of the weights. The 2.75 GB is consistent with a 3.81B model and does not independently confirm one.
Bytes moved per token is the arithmetic that tracks decode speed. The pruned model reads all 2.75 GB every token. The full 26B, at the 4.6 bits per parameter implied by its 14.5 GB footprint, reads only the 3.8B active parameters, about 2.2 GB. The small model is roughly a quarter more expensive per token, not cheaper. Wilber's chart has the full 26B at 94.1 tokens per second against 79.0 for the pruned model, which he attributes to the different runtime. The two-times claim in the post is against the 12B, at 41.3.
A method you can rebuild, a result you cannot
None of this came from the Gemma team. Wilber is one developer with a 48 GB laptop and a product, Tolby, that needed a classifier resident all day. What he did publish is the Shears toolkit: MIT-licensed Swift on MLX, with a CLI, documentation and a Colab notebook, and an export path verified byte-identical to llama-quantize tensor by tensor.
So the technique is reproducible and the demonstration is not. The classifier's expert selection, prompts and weights stayed private. The translator's training pairs were withheld, its teacher was a 4-bit QAT checkpoint rather than Google's bf16 model, and its numbers come from one run, one seed, no human evaluation.
The defensible claim is narrower than the headline. A model in the 3.8B band can be made to hold one narrow behaviour at roughly its 26B teacher's agreement level, for about 76 minutes of laptop training. Whether it holds that behaviour when the prompt changes, the input shifts or an attacker arrives is not in the post, and the author says so himself. The 85 percent is the least interesting number in it.
