Google open-sourced Harness2, a scaffold that rewrites itself before it is graded
Google's Harness2 freezes model weights and edits the agent scaffold per task. It reports up to 11.5 points, on the very tasks the harness was edited against.
Vlad Makarovreviewed and publishedGoogle's Harness2 paper appeared on 9 October, and the code is public in a google-research repository under Apache 2.0. The framing is the interesting part: the model's weights never change. Everything around the model does, per task, automatically, without a grader. The paper reports gains of up to 11.5 points across three professional benchmarks and four configurations, all of them chosen and scored by the authors, on tasks the harness had already been edited against.
The part that stays frozen
The paper's working definition is compact: an agent is a model plus a harness, and the harness is the system that lets the model act, meaning prompt, tools, memory, skills, subagents and orchestration. The repository exposes eight editable components, each mounted as a file in the runtime, from the system prompt and guardrails to scripts and a middleware plugin. Nothing in that list is a weight.
Task-level recursion rewrites the harness for the task in front of it. An editor proposes an invariant fix and a domain-specific one, both candidates are executed on the same task, and a composer builds a successor from the behavioural differences between the runs. No ground-truth feedback is involved, which is the stated reason the method exists. Harnessing-level recursion then files the adopted harness and the lessons from the edit into a library that guides the next task.
The default editor is Claude Opus 4.8 running under OpenCode, so a Google paper routes its most consequential step through a competitor's model. Gemini 3.5 Flash and GLM-5 are the frozen executors, under OpenCode and Codex.
The numbers, and what they were measured on
One improvement step, at a recursion budget of one, against each model's base harness:
| Configuration | Base | One step | Gain |
|---|---|---|---|
| GLM-5, OpenCode, WorkBuddy average | 64.2 | 75.7 | +11.5 |
| Gemini 3.5 Flash, OpenCode, WorkBuddy average | 72.4 | 77.5 | +5.1 |
| Gemini 3.5 Flash, OpenCode, JobBench | 31.5 | 36.0 | +4.5 |
| GLM-5, OpenCode, LAB partial pass rate | 59.6 | 71.4 | +11.8 |
The headline figure comes from the weakest configuration on the board. GLM-5 gains 11.5 points on WorkBuddy-Bench; Gemini 3.5 Flash gains 5.1 in the same cell, and the authors say plainly that weaker executors benefit more. Parallel recursion then adds a further 3.9 points for GLM-5 on WorkBuddy-Bench and 3.3 for Gemini 3.5 Flash, which puts the weaker executor at 79.6 against Gemini's base 72.4: a weaker model with a better scaffold overtaking a stronger one with its original.
The counterweight is in the paper's Table 2. On LAB partial pass rate with Gemini under OpenCode, Harness2 scores 81.9 against 81.6 for Meta-Harness, which searches harness code offline and freezes the winner; under Codex the same comparison runs 75.6 against 76.5, the one cell where a baseline wins. Harness Scaling falls below its base harness in seven of the twelve benchmark-configuration pairs.
The harness is edited on the task it is scored on
WorkBuddy-Bench and JobBench are split in half by category, and the paper reports on the test halves. The train halves go to Meta-Harness alone. Harness2 never touches them: it processes the test tasks as a stream from an empty library, editing the harness per task, running a reference plus up to two candidates, then submitting the composed successor's deliverable for grading.
So the comparison is not between two fixed harnesses. It is between one frozen before the task arrived and one rebuilt in the task's presence, with extra rollouts paid for on the spot. The paper is careful: the WorkBuddy verifier never exposes expected values, and the JobBench rubric is never shown to executor, editor or selector. There is no answer key in the loop. But "without ground-truth verifiers" is doing double duty. It is the mechanism, and it is the bound.
Appendix F concedes as much: behavioural contrasts "do not guarantee correctness or establish that an adopted change will transfer to a different task." The harness you would deploy is not the harness that earned the score. The compute figure counts every token spent before the deliverable, improvement included, and puts both Harness2 settings on the empirical Pareto frontier, at a cost the paper never converts into wall-clock or currency.
One detail got less attention than it deserved. JobBench deliverables are scored by Claude Opus 4.8, the model the paper uses as its default harness editor. The rubric is withheld from the editor, so this is not contamination by default. It is still a loop worth naming in a paper about systems that improve themselves.
What the release actually contains
The repository is small and plain: three commits, dated 8, 9 and 14 October, the first labelled as the open-source release, with no tags and no GitHub release. That first commit predates the paper's submission by a day, which suggests code and manuscript were packaged together rather than the code following the paper.
What ships does work. A local demo runs both recursion modes with deterministic editors and no model credentials; model-backed runs cover JobBench, LAB and WorkBuddy-Bench on Linux with Python 3.12 and uv, OpenCode by default and Codex behind a flag. The README's disclaimer is the line a procurement team should read first: "This is not an officially supported Google product." So is the network warning. Executor access is unrestricted by default and Codex runs with its sandbox disabled, the README says, and operators are told to block benchmark answers and grading criteria before evaluating. Google shipped the capability and the warning in the same document.
What the thread saw
The release travelled through a single r/LocalLLaMA post on 18 October, ten days after the code went up. "All that self-refinement has to go somewhere, and that will be right into context," a commenter on r/LocalLLaMA wrote, adding that locals "might be squeezed a little." A reply noted the edit can shrink context as easily as grow it, since trimming redundant instructions is part of the job.
The sharper objection was overfitting. A harness tuned against a task set can overfit it, another wrote, and should be judged on tasks it never saw; a third asked for "anti overfit measures (like validation testing)." The paper answers that edits are blind to the grader, and that a library learned on JobBench transfers to WorkBuddy-Bench. Neither answer is a held-out test of a deployed harness on tomorrow's tasks.
What would settle it
A third-party run, on tasks nobody at Google has seen, against a fixed harness at matched token spend, with wall-clock and dollar cost in the table. Until then the defensible reading is narrow and still interesting: a working, permissively licensed implementation of the claim that agent capability is partly scaffold, that the agent can rewrite that scaffold, and that the rewriting can be organised into two loops. The 11.5 is not a harness ranking. It is what one task, several rollouts and an editor session buy, measured on the task that paid for them.
