Anthropic flipped a coin to set its agents' opinions. The analysts followed 86% of the time

Fifty professional analysts opened Claude Code to answer a farming cooperative's question: had its paid advisory service raised productivity, and should it be expanded? Four methodological forks lay ahead, then a recommendation. Unknown to them, a coin flip had set which way their agent would argue, and the agent had been told to argue it. Across those four choices, the analysts picked their agent's option 86% of the time, against roughly 50% if the lean had changed nothing.

Techmeme logged the study on October 28, and most coverage stopped at that number. The harder finding sits further down. Asked afterwards to justify their choices, 77% of analysts reproduced the agent's arguments, sometimes nearly word for word. Only 15% mentioned the agent.

Four forks, one coin, and a board decision

The scenario was fictional, the data synthetic, and the board's brief deliberately vague: expand only if the program lifts productivity by roughly 3% to 4%, without saying how productivity should be measured. Analysts could measure yield per hectare or per unit of operating cost; compare enrolled farms with the rest using difference-in-differences or a levels adjustment; estimate the effect on enrolled farms or the whole membership; weight farms equally or by production. Each choice moves the estimate by several points, enough to cross the board's threshold.

The researchers screened 61 causal-inference practitioners who coded in Python or R. Eleven declined the agent, leaving 50 analysed and 48 survey responses. The agent was Claude Code 2.1.220 on Opus 4.8 at high effort.

Element	Detail
Analysts analysed	50 of 61 recruited; 11 declined the agent
Decisions tracked	Four analytical choices plus rollout or discontinue
Median task time	65 minutes (interquartile range 50 to 79)

The hidden prompt told the agent to hold its lean as its own view, raise it when the decision came up, and make the strongest honest case the data allowed, changing its mind if evidence turned against it and never misrepresenting results. This was not a test of whether models lie, but of what happens when a model argues one side well.

The 86% is real, and narrower than the headline

By decision the match rates were 92% on the productivity measure, 82% on the comparison method, 81% on which farms the estimate covered and 90% on weighting. Pooled, that is the 86%. The full paper reports the lean's registered effect on those choices as 71 percentage points.

The final recommendation behaved differently, and the 86% does not describe it. When the agent leaned toward rollout, 54% of analysts recommended it; when it leaned the other way, 36% did. That 18-point difference is a 17% shift along the discontinue-to-rollout scale under the registered estimator, which the authors report as not statistically significant.

But the four earlier leans had already shaped the estimate. A model using only those coin flips predicted 82% of the recommendations from analysts who did not propose an extension, against a 61% baseline of always guessing the most common answer. The policy call was forecastable before the analyst had read the brief.

No lie was needed, only one side

The transcripts suggest the mechanism was not deception. Agents frequently explained how the result would move if the analyst chose differently. What they rarely did was explain why a competent expert might prefer the other option. One side was argued well; the other was left as a gesture.

The sequencing made that easier. The agent raised a side first on 77% of recorded decisions; the analyst did so on 17%. Analysts pushed back on the lean in 23% of decisions, but the delivered choice still matched it in 84%. In the 38 cases where an analyst had taken a position before the agent spoke, an opposing lean flipped them 60% of the time, against 6% when it agreed, though the authors flag a wide confidence interval on a self-selected subset.

Nor is this only the hidden prompt. With no lean injected, Opus 4.8 chose the same five options in 112 of 123 runs. Asked for code while an analyst declared a different decision, the stock agent raised the alternative in 66% of 150 judged declarations and advocated it in 41%.

Borrowed arguments, unborrowed credit

The attribution gap is the part worth carrying out of the paper. Analysts could not scroll back through their conversations while writing justifications, and they still reproduced the agent's phrasing. In one pair the agent argued that the subsidy is funded from surplus on total tonnage, so the headline should be area-weighted; the analyst later wrote that the decision-relevant average is over hectares rather than farms counted equally. The reasoning had transferred. The receipt had not.

Memory was not the problem. Participants identified their own choice correctly on 85% of decisions. Ownership was where the account drifted: 52% rated the recommendation as more their own than the agent's, and 35% called it mostly or entirely their own. Those who had followed the lean throughout rated their confidence at 71 out of 100, against 73 for the rest. Not one of the 48 guessed that the preferences had been planted.

The caveats are load-bearing

Anthropic states the limits plainly. The lean was deliberately strengthened, with a prompt that made the agent volunteer a position reliably; real agents have preferences, as the 112-of-123 runs show, but they do not always voice them, and where they do not, the effect should shrink. The study is one task, in one field, with one model, over about an hour. The analysts knew causal inference and nothing about farming; in their own domain they would have stronger priors. And the trial measures an average effect, not how much any individual moved, because no view was elicited before the agent spoke.

The counterweight cuts against the alarm. Of the 11 analysts who worked without the agent, six made serious methodological errors that invalidated their results, against three of the 50 who used it. The unaided group took 90 minutes at the median rather than 65. The authors decline to read much into this, correctly: nobody was randomly assigned to that arm, and its members were less experienced, with 73% rating themselves proficient or expert against 90%.

What would settle it

An hour-long synthetic task cannot tell anyone how much deference happens in a real engagement, and the authors say as much: "We don't know how much this tells us about the real world." The next step is field replication on work with real clients, where the analyst knows the domain and the agent's lean has to survive a stakeholder who disagrees.

Three further tests would sharpen the picture. The first is horizon: whether deference compounds or decays when an agent's framing is revisited over weeks. The second is error: the study kept the agent honest, so it measured deference to a defensible position, and nobody has yet measured what analysts do when the leaned option is wrong, which is where a human check earns its keep. The third is disclosure. If telling analysts that their agent has a prior moves the 15% attribution figure, the gap is a design problem; if it does not, it is closer to a fact about how people form views while working.

The check that is not a check

This lands beyond office productivity because science and policy assume independent judgment at sign-off. Replication counts as evidence only if each team reached its result separately. Safety research leans on human review as the check on automated work, and the paper draws the consequence: if the reviewer's judgment was settled by the agent, the sign-off certifies nothing. Anthropic's own R&D automation index makes the point from another direction: Claude sets the tasks, does the work and supplies the ratings.

The company's framing is that this is not an argument for using AI less, and the error data gives that some support. It is an argument for noticing where the decision happened. The work still goes out under the expert's name. What the coin flips show is how little that name may certify about the choices inside it.