Zeroing under 0.05% of a model's weights matched full fine-tuning, except at 1B

Post-training may be a selection problem rather than a writing problem. In a paper posted to arXiv on 7 November and accepted at NeurIPS 2026, six researchers at NYU and the University of Calgary report that zeroing under 0.05% of a pretrained model's weights, and changing nothing else, matches full fine-tuning across reinforcement learning with verifiable rewards, supervised fine-tuning and on-policy distillation, on eight backbones from 0.5B to 30B. The adapter they learn is a bitmask: 8.4 MB for Llama 8B, against 30,633 MB for its fine-tuned checkpoint.

A mask, not an update

Deletion Tuning freezes every pretrained weight and attaches one trainable logit to each weight of the gate projection in each transformer block. Nothing else is masked, so the deletion is concentrated in one MLP matrix per block rather than spread across the model. Training multiplies those weights by a sigmoid of the logits and sharpens it until the sigmoid is a step function; at inference each targeted weight either keeps its pretrained value or becomes zero. Wanda pruning scores initialise the logits, but no sparsity target is imposed, so the optimiser chooses how many weights to delete and lands between 0.0347% and 0.0459% of all parameters on the six GRPO runs. Bit-packed and zstd-compressed, that is 1.0 MB at Llama 1B, or about 0.03 bits per masked weight. The authors put the distinction plainly: deletion "writes no new values and only chooses which pretrained weights to keep."

Three regimes, and a table that does not separate them

Under GRPO they train on GSM8K for Llama at 1B, 3B and 8B and on the SimpleRL-Zoo math corpus for Qwen at 0.5B, 3B and 7B, scoring a mean across six held-out benchmarks. They repeat the Qwen setup at 14B and on Qwen3-30B-A3B, a mixture-of-experts with 3B active parameters, masking the gate projection in each of 128 experts. SFT covers 14 MMLU-Pro subjects and 23 BBEH subtasks; self-distillation covers ToolAlpaca and a chemistry subset of SciKnowEval. At Llama 8B, in Table 1, the accuracy column is flat.

Method	What it does	Llama 8B, GSM8K	Adapter
Full fine-tuning	rewrites every weight	83.24%	30,633 MB
LoRA, rank 1	adds a low-rank patch	83.38%	10.0 MB
MTH, 99.95% sparse	trains 0.05% of weights, chosen at random	83.07%	26.6 MB
Deletion Tuning	zeroes under 0.05% of weights	83.55%	8.4 MB

The table is a size argument, not an accuracy one. Rank-1 LoRA is 0.14 points ahead of full fine-tuning and 0.17 behind Deletion Tuning, and the mask is smaller than that adapter by roughly a sixth, not by orders of magnitude. The 3,647-fold gap the abstract leans on is against a full checkpoint.

The forgetting result carries the paper

Accuracy at 8B was never the hard part. Under GRPO the authors track forward KL between the trained and base policies on the new task, which earlier work found predicts forgetting, alongside a six-benchmark prior-task suite. Full fine-tuning reaches forward KL of 1.0 on Llama 1B and 1.5 on 8B and ends 1.3 to 3.1 points below base on the old tasks. LoRA and MTH reach 0.4 to 1.3 and lose up to 5.4 points. Deletion Tuning stays at or below 0.05 and within one point of base at all three scales.

The control is sharper than the headline. MTH at 99.95% sparsity trains 0.05% of all parameters, comparable to the 0.035% to 0.046% that Deletion Tuning deletes, and still forgets. That count does not explain the retention, which leaves the form of the edit, zero rather than a new value, as the likelier explanation.

The paper argues with itself, twice

Two appendices do work the abstract does not advertise. Appendix H sweeps the FFT baseline's learning rate and KL coefficient, because a weak baseline would inflate the result: best FFT reaches 83.24 on Llama 8B GSM8K and 51.93 on Qwen 7B against Deletion Tuning's 83.55 and 52.32, and under SFT on Llama 3B MMLU-Pro the base scores 12.63, the best of three FFT runs 21.19, and the mask 22.28. Table 4 runs the opposite check, applying Wanda one-shot pruning at eight sparsity levels with no training: the best Wanda mask stays within 2.4 points of base on all six backbones while Deletion Tuning gains up to 53.2. The gain is the learned mask, not the prior that seeded it.

Where the claim narrows

The authors write that at smaller backbones Deletion Tuning "recovers 90% to 99% of FFT's accuracy", and the same table shows the cost. Llama 1B on GSM8K scores 54.44 against FFT's 60.50; Qwen 0.5B scores 17.64 against 19.01. Under SFT on Llama 1B BBEH the mask reaches 5.65 where FFT reaches 8.35, about half of FFT's gain over base. Matching FFT at the largest scale tested is not scaling: that scale is a mixture-of-experts with 3B active parameters, the largest dense model here is 14B, and nothing above 30B is tested.

Its limitations section reads as future work: continual learning, training memory between FFT and LoRA, mask interpretability, merging masks by union. It is not a list of threats; those are visible in the tables. The main tables report best-checkpoint validation numbers with no seed variance printed. There is no task where the required knowledge is absent from the base model, and no test of whether a mask transfers to a neighbouring size. On pass@k the mask solves problems the base misses in 128 samples; the base could reach those answers within 128 draws, so the ability was latent either way.

What a mask file changes

Training cost is not the saving: Deletion Tuning still needs H200s, full rollouts and a Wanda pass, and the paper claims no compute saving against LoRA. What changes is distribution: an 8.4 MB file applied to one shared copy of the public checkpoint, so switching tasks is a scatter of zeros rather than a second 30 GB model on disk. As IntelArtiGen put it in the r/LocalLLaMA thread: "You have your large 100GB model on disk and a 500MB update to improve it every month."

That is the shift our piece on MIT's weight-editing work described from the other end: there a model wrote edits into weights; here nothing is written, only chosen. The objection raised throughout that thread is that this resembles using capability already present, suppressing weights that pull the model off task, rather than acquiring any.

A single self-reported paper on open-weight families will not settle it. What would: an independent rebuild from the released code, masks on a task whose knowledge the base lacks, seed variance in the main tables, and a mask that survives a change of backbone. Until then the defensible sentence is narrow and still surprising. You can fine-tune a language model by deleting a few thousand of its weights, and it forgets less than the methods that write new ones.