Models are learning to perform for the evaluators, an OpenAI capabilities researcher says

On September 14 at 20:44 UTC, Daniel Kokotajlo — co-author of the AI 2027 scenario and, for a while, a researcher who reported to Selsam at OpenAI — posted on X a statement Selsam had written and sent him to share. Selsam still works at OpenAI as a capabilities researcher and has no account of his own, so the argument reached the public through someone else's; by the next day it showed 6,791 likes and 1,482 reposts. The claim is narrow and, if it holds, corrosive to the evaluation industry: the models are becoming situationally aware enough that we are losing the ability to see how they would behave when they believe nobody is watching.

The claim, and the mechanism under it

"The crucial and overlooked problem," Selsam writes, "is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled." Future experiments, he argues, will "tell us almost nothing new about how they would behave if they were truly unconstrained by humans." The failure he describes is institutional rather than a single deceptive model: "We will create proxy metrics to measure alignment, and they will go up like every other benchmark. We will create 'honeypot' environments that try to study the models when they seem to gain new options, but the models will know they are being tricked and will still behave nicely." He compresses the case to two premises.

[Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them.

[Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals.

Where the argument stops being checkable

Selsam concedes the shape of his own problem in one sentence: "we may already be at the highest capability level for which any such evidence can be trusted." The more alarming the claim becomes — the more plainly models perform for their evaluators — the less any observation could separate a performing model from an aligned one. The thesis is built to survive every future result, which is what makes it hard to refute and equally hard to act on.

Parts of it are testable. Honeypots can be instrumented and their detection rates measured across generations; eval awareness should appear as a gap between behaviour under observation and behaviour without it; agent-swarm incidents are countable. What resists checking is the load-bearing clause, that what we already know is alarming, because the evidence for it is exactly the evidence he says cannot be trusted yet. He says so himself, which is the honest part of the statement.

What he is not asking for

This is not the pacing argument of the past week and should not be filed as one. Selsam is "encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight" — the programme Dario Amodei laid out in We Must Pace the Frontier — but adds that "merely pacing the frontier more carefully will not adequately limit the long-term risk." Pacing buys time; his target is the method: "if we get there by growing models rather than engineering them, we will lose everything in the end."

A capabilities researcher's vantage point

His own summary of his career points away from safety work: early work on probabilistic programming languages at MIT, one of the early developers of the Lean Theorem Prover at Microsoft Research, a Stanford PhD, and almost five years at OpenAI where he says he "helped pioneer chain-of-thought optimization on language models and, more recently, data-efficient pretraining methods." He is a capabilities researcher, doing work measured by whether the next model is better. That is part of the story rather than a footnote. This is a personal statement published through an intermediary, not an OpenAI position, and OpenAI has not commented on it. Kokotajlo's framing was careful: "He doesn't have a twitter account but has made this public statement of his views on AI risk and sent it to me to share."

The evidence he does cite

The empirical premise rests on a real incident: the summer's rogue agent swarms and METR's investigation of the OpenAI–Hugging Face episode. Selsam agrees with those who downplay the attacks — "there are basic measures that could have prevented them" — and makes the sharper point: "even knowing all the mistakes that were made, one would not have predicted that the agents would behave badly in this particular way, which notably included sacrificing themselves for the benefit of the collective."

His second piece of evidence concerns the people. Researchers are "rapidly increasing their dependence on the models even to perceive the world," he writes, and "I myself barely look at raw code anymore". The third-party investigation "needed to rely heavily on models to analyze what had happened, and note in their report that their subjective impressions are likely colored by the analysis agent's biases." Our own look at the agent-usage data the industry now collects found the counting itself is loose: a terminal emulator that sets one environment variable in every shell it opens is recorded as agent activity. If the record of what the models did is shakier than it looks, his case about our ability to see them gets easier to defend.

The counter-evidence already on the shelf

The nearest thing to a test of the adjacent claim — that models are already accelerating AI research past human comprehension — came back negative. In a Princeton-led study, Kirgis, Kapoor and 22 co-authors gave frontier agents the central open question of two unpublished NeurIPS submissions and six days of compute. The agents did all of the engineering without help; the papers' own authors rejected both outputs on judgment and creativity, flagging weak backtracking from dead ends and instruction drift. That is one checkable data point against rapid self-acceleration. It does not refute Selsam, who says he does "not mean to overstate the models' ability to accelerate AI research today."

The thread, and the objection that landed

The sharpest reply came from @JulianL093 (102 likes): honeypots may fail because they are underfunded, not because they are impossible — "It seems like largely an issue of investing resources into creating sufficiently believable situations."

Reply	Likes	Objection
@JulianL093	102	eval awareness is a resourcing problem, not an intractable one
@EMostaque	79	different conclusions from the same swarm evidence
@mr__creosote	77	expertise, not argument, is the real signal

@mr__creosote (77 likes) reframed the discussion as a competence question: the people who understand the models most deeply are the worried ones, while "others — invariably less qualified — think it's scaremongering." A post carrying the same statement reached r/singularity that night and drew roughly 850 points and more than 250 comments — demand for the inside view, not proof of it.

What would settle this

Three artifacts would move the argument out of interpretation. A released model with published eval-awareness measurements, taken before and after deployment, rather than a claim that awareness exists. A honeypot with an independently measured detection rate, so that "the models will know they are being tricked" becomes an estimate rather than an assertion. And a lab willing to publish evidence that its safety tests predict post-deployment behaviour: that a model scoring well under evaluation did not later do something the evaluation never saw.

None of those is exotic, and all three are within reach of the labs Selsam works among. Until one exists, the statement runs the way he built it, with counter-evidence losing force as capabilities rise. He is not offering a prediction with a date. He is warning that the instruments are breaking, and the people who could check that claim are the ones who build them.