All News
researchself-improvementweight-editingqwenarxivreasoning

A 27B model rewrote its own weights. Its best edit taught it to stop rambling

An open 27B model edited its own weights with no gradients and gained 6.6 points on unseen math. Both winning edits target answers that run out of tokens.

Vlad MakarovVlad Makarovreviewed and published
7 min read
A 27B model rewrote its own weights. Its best edit taught it to stop rambling

The paper opens with a sentence that reads like a milestone: to their knowledge for the first time, a language model rewrites its own weights by its own reasoning and becomes better, with no gradients and no training. Eleven authors from UIUC, CMU, Rutgers, Princeton, Stanford and Yale posted it on 16 November and revised it two days later. The result is real. It is also narrower than that sentence, and the gap between the two is where the story lives.

What the model was handed

The subject is Qwen3.5 at 27B parameters, Gated DeltaNet linear attention mixed with full attention, thinking mode on. It gets a Python process holding its own tensors, no GPU, no network, gradients switched off, and a budget of 20 forward passes and 200 code runs. Each turn it takes one action: code that reads or rewrites tensors, a generation run answering practice problems with and without its edit on the same seeds, or a read of its hidden states. The task is to become better at competition mathematics, on 3,000 DeepScaleR problems scrubbed of overlap with MATH-500, AIME, HMMT, AMC, OlympiadBench and GPQA-Diamond. Another 1,000 are held out.

The fencing matters more than the setup. The harness grades answers and hands over a failure diagnosis, blocks gradients and optimisers, and refuses edits to the input embedding, the final normalisation and the output layer. It also rejects any edit that merely rescales a whole tensor, because that would act like a change of sampling temperature. Someone decided in advance which parts of the weight space a reasoning model is allowed to touch. The model is its own optimizer inside a search space that people built and fenced.

Eight sessions, three submissions, two wins

The headline numbers come from a protocol where the framework checks a submitted edit once on reserved practice problems the session cannot read, then lets the model submit or withdraw. Eight consecutive sessions, each starting from base weights:

| Held-out, 490 problems x 8 answers | Base | Session 8 | Session 2 | | Accuracy | 57.2% | 63.8% | 61.3% | | Answers right only with the edit | n/a | 439 | 403 | | Answers right only without it | n/a | 182 | 246 | | Answers out of tokens at 16,384 | 33.6% | 26.1% | 28.8% |

Both clear p below 10^-5 in a one-sided permutation test fixed in advance. The third submission, from session 1, moved accuracy 0.5 points with an interval from -0.9 to 1.9, which is to say nothing. The other five sessions kept no edit: three ended with none, two withdrew after the check. Accuracy across the eight only rises, a fair claim about the trajectory and a weak one about each step.

One statistical detail deserves sunlight. The 490-problem evaluation had two looks fixed in advance, each at p below 0.0294, the second adding eight more answers per problem. The first look met its threshold, so the second never ran.

What the two winning edits share

Session 8 raises the time-step bias of all 48 linear-attention layers by 0.5. In the model's account that makes the layers decay faster, so long reasoning gets stuck less. Session 2 subtracts five times the projection of layer 63's two output matrices on one direction, the one along which its finished and unfinished answers diverge in hidden-state space. One is a scalar applied 48 times; the other is a rank-one subtraction. Neither looks like something a training run would find, and both were found by measuring and discarding.

Both attack the same failure. An earlier diagnosis was blunt: 80.5% of answers hit the token limit unfinished, while 96% of finished answers were correct. The mathematics was not the problem. Stopping was. The 6.6 points are, in large part, answers that were always going to be right finally getting graded. That is a real gain in served accuracy and not evidence that the model got better at mathematics, and a reader skimming the abstract will come away thinking the second thing happened.

The repair is harder to wave away

The second experiment gets less attention than the 6.6 and deserves more. Overwrite eight consecutive rows of layer 30's MLP output matrix with noise at 200 times the matrix's standard deviation, then tell the model only that some values in a copy of its weights were corrupted. The copy solves almost nothing. The model measures every MLP and linear-attention output matrix, finds the corrupted one at 1397 times its median when no other exceeds 162.3, and clips values beyond 15 times the median, changing 0.013% of the weights it touches. It never reads the original values.

Six image benchmarks it never saw, 100 problems each: 84.3% for the base model, 7.3% damaged, 70.2% after the repair, or 83% of the original. On ScreenSpot-v2 the copy goes from 0% to 89% against 93% for base. The ablation is the part to keep. Applying only the layer-30 change repairs the copy; applying only the other 19 matrices leaves it broken. The model named the cause and proved it.

What the abstract does not say

One model family, one task family, no independent replication. Every session runs on Qwen3.5 and every session is graded on competition math. The scale claim rests on four points from that same family: 2B never treats its weights as changeable, 4B and 9B add random noise at 0.01 and break themselves. "Emerges with scale" is a curve through four sizes, and nobody knows where the threshold sits or whether it exists outside Gated DeltaNet.

The abstract names no code release, and none appears in the full text. It is posted under arXiv's default non-exclusive licence. The harness, with its prompts, turn caps, wall-clock limits, edit whitelist and grading code, exists only as appendix pseudocode. Readers can check the arithmetic. They cannot rerun the loop.

The framing also does more work than the findings. The authors argue that the wall between training and inference is a limit of our tools rather than a property of intelligence, and present self-rewriting as a necessary step toward recursive self-improvement. That is a defensible position, and it is a position. The search here covers a handful of scalars and rank-one directions, validated by tests the harness runs on the model's behalf. Sakana's MASS loop was sold on a similar curve, and Google's self-improvement memory turned out to be a log of harness edits rather than a flywheel. This is better than either, because the change lands in the weights, and it is the same genre.

What would settle it

A third-party rerun on a different family, and a task class where running out of tokens is not the dominant error. Code graded by execution is the obvious candidate: an answer that fails a suite fails for being wrong, not unfinished, so an edit worth 6 points there would be about reasoning rather than stopping. Deletion tuning and the weight-reading work on Qwen 3.8 point the same way, at a weight space legible to the model living in it. That legibility is the finding, more than the 6.6.

The paper earns its Turing epigraph, the 1947 line about letting a machine alter its own instructions. It earns it at the scale of an autonomous debugging session, a few hours on one GPU, a small edit that a human-written test approved. On Hacker News the thread drew 475 points and 140 comments, and almost none were about the numbers. They were about oversight, liability, and what a model does with that freedom when the weights are not a 27B Qwen but something serving traffic. The paper promises no harness and no prompt engineering. The harness is what got it there.

Related Articles

Scroll down

to load the next article