METR measured a 2x speedup at Anthropic. Most of the evidence arrived redacted
METR's eleven-week audit puts Anthropic's capability growth near twice its old rate, and concedes Anthropic withheld most of the evidence behind that number.
Vlad Makarovreviewed and published
Eleven weeks, 2.5 full-time staff, and access to internal data no outside evaluator had seen: METR's assessment of AI R&D acceleration at Anthropic, published on 5 November, is the closest look anyone has had at whether a frontier lab is now speeding up its own research with its own models. The headline is that as of October 2026 Anthropic is "likely improving the capabilities of their models on well-specified tasks at around twice the rate they did between March 2024 and October 2025", a figure METR immediately widens to a plausible 1.3x to 3x.
The sentence that deserves equal billing sits at the top of the page, in METR's redaction summary: "Anthropic redacted a majority of the evidence we found most informative for these conclusions."
What the access actually consisted of
The window ran from 12 August to 29 October. METR could request anything relevant and Anthropic could refuse unilaterally; in practice, METR writes, the access "surpassed our expectations". It received internal benchmark scores and AECI values for unreleased models, algorithmic efficiency histories in pretraining and RL, training compute by domain, headcount with progress attributed to staff, lines of code, agent inference spend, and roughly fifteen interviews. The team worked on Anthropic laptops and took no payment.
Two limits belong next to that list. METR did not request or receive query access to an unreleased model, and its information ran through two technical staff, which it calls an adversarial robustness weakness. More awkwardly, the project was originally contracted to verify how Anthropic splits compute between capabilities and safety. Anthropic changed the scope midway and the component was dropped "before METR came to any substantive conclusions on compute usage at Anthropic". The safety share is a figure Anthropic reports about itself; the audit of it did not happen.
The numbers, and how far they bend
| Estimate | Value | Scope |
|---|---|---|
| AECI acceleration since Oct 2025 | 1.9x (95% CI 1.5 to 2.8x) | Anthropic models, internal snapshots included |
| Overall capability acceleration | about 2x, plausibly 1.3x to 3x | METR's subjective judgement |
| Share attributed to AI | perhaps 80% | October 2026, substantial uncertainty |
| AECI, all frontier developers | 1.4x | adds OpenAI, Google and xAI models |
| Same data, different link function | 1.1x to 2.9x | depends on assumed benchmark difficulty |
The AECI series, Anthropic's version of Epoch's capability index, ran at about 15 points a year before October 2025 and about 29 after it, with the breakpoint estimated at 7 October 2025. That is roughly two years of historical progress in one, the equivalent of travelling from Claude 2.1 to Sonnet 4.5 again in a year. On training compute METR is unhedged: "We are confident that the acceleration was not due to an acceleration in the scaling of training compute." The evidence for that confidence is redacted. The public gets conclusions and not the workings.
The 1.9x also sits above the 1.7x in Anthropic's Haiku 5.5 system card, a gap METR attributes mostly to including models the public has never used.
Two readings of the same curve
Take the same benchmark scores and widen the sample to every frontier developer, and acceleration falls to 1.4x, from 17 points a year to 24. METR flags the confound itself: "the historical trend is estimated mostly from non-Anthropic models, while the current trend is estimated from mostly Anthropic models." Two interpretations follow. If Anthropic was subfrontier in 2025, its historical rate was understated and 1.9x is too high; if its models were less benchmark-optimised than rivals, the historical trend was overstated and 1.4x is too low. METR publishes both and picks neither.
The sharper problem is statistical. AECI converts benchmark scores into a latent capability score with a logistic curve, which embeds an assumption about how many tasks in a benchmark are broken or guessable. Use a complementary log-log link instead, assuming a long tail of easy tasks, and the same data yields 1.1x, with an interval of 0.8 to 1.5 that is compatible with no acceleration at all. A log-log link, assuming a long tail of hard tasks, yields 2.9x. Both fit about as well: 87.1% against 86.9% out-of-sample R-squared. METR writes that "in practice most functions fit the data about equally well", and keeps the logistic link because Epoch and Anthropic use it by default. That spread, from rounding error to near-tripling, is the story's centre.
What the public cannot check
The report's table of shared and withheld evidence makes the asymmetry legible. METR was given key intermediate capability measures, algorithmic progress estimates, most R&D input measures, and details of bottlenecks and forecasts. None of those four categories appears in the public write-up. The row for AI's contribution reads "Partly", and behind that word sit per-stage attribution estimates for pretraining and RL.
Which is why the number that travelled is not one of them, and why the transparency question has teeth: METR's own policy says a report should let the public judge the evidence itself. The r/singularity thread that pushed the number around carries a title saying Model 4 "scored 178.6 on the AI capability index, a jump nearly 90% as large as the entire improvement from Opus 4.6 to Mythos 5". The 178.6 is real, and it appears in METR's report exactly once, inside a diagnostic: three of fifteen benchmarks had notably flat or concave fits, and dropping them raises Model 4's score "from 178.6 to 179.7". The report prints no per-model scores for Opus 4.6 or Mythos 5, so the 90% comparison is eyeballed off a chart, not stated anywhere. The screenshot attached is a different figure: the one arguing a permanent slope change predicts held-out models better than a one-time jump.
Model 4 finished training in September or October 2026 and scored above 50% on all fifteen benchmarks used for it. METR reads that as its own problem: "This lack of benchmarks which are difficult for Model 4 makes us somewhat nervous about our ability to measure AECI now and in the future."
One more disclosure. METR's second adversarial robustness weakness is reliance on Claude, involved in most of the data collection and analysis it received and in "the majority of our own analysis". The evaluator's instrument was the evaluated company's product. METR found no discrepancy against human interviews, which is a mitigation, not a null.
The evidence that survived
Interviews are the least redacted channel. Of eleven researchers coded from interview notes, two clearly endorsed a doubling or more in their area, three leaned that way, four were unclear, two said no, and none leaned no. METR calls the result "mildly in favor of 2x or greater acceleration". One dissent, paraphrased in the notes, put AI's contribution at 15 to 20% and dismissed self-reported productivity gains outright.
The internal Slack polls, a cousin of the R&D automation index Anthropic published in September, tell a louder story: average self-reported uplift reached about 12x for the most capable internal model in October 2026. METR then takes its own evidence apart, citing low-effort reactions, an unrepresentative subset of staff, and its own 2025 study finding that developers overestimate AI speedups. On autonomy the report is careful: AI cannot yet make fully autonomous contributions to the most important parts of training, but researchers described "signs of life" that it soon might.
What would settle it
The ~100-page report behind the public one exists and sits with Anthropic. Publishing the per-stage attribution, bottleneck analysis and forecasts would let outsiders do the reconciling METR says it cannot. Epoch's index code is public, but the input it needs, Anthropic's internal benchmark scores, is not, so independent reconstruction of AECI runs on the same redacted evidence or not at all.
The rest is arithmetic that one company cannot supply. Restart the compute-allocation verification. Run the same eleven-week audit at a second lab, with the right to describe what was seen. Until one of those happens, the honest summary is METR's own: the public has to rely on the evaluator's judgment "much more than would be ideal".
