Artificial Analysis rebuilt its index on private tests, and v4 scores no longer subtract
Artificial Analysis Index v5 puts 60% of the score on private test sets, widens the open-weights gap to 12 points, and ships another index to span that gap.
Vlad Makarovreviewed and published
Artificial Analysis published Intelligence Index v5 on October 27, and the first thing to understand is that this is not a rescoring. Two evaluations arrived, three were replaced or revised, three were dropped, and 60% of the composite now rests on test material the public cannot see. The second is that Claude Opus 5.5 (max) leads at 58, one point ahead of GPT-6 Astra (max) and three ahead of GPT-6.1 Sol (max), with Gemini 4 Argon (high) next at 50. The third is the one that matters for anyone keeping a spreadsheet: those numbers cannot be subtracted from last week's v4.3 scores, and the people who built the index say so in writing. That admission is why the same announcement shipped a second index built to survive exactly this kind of revision.
What changed, and how much of it you can't see
Both additions are agentic and built to resist memorisation. FrontierCode 2.0 is Cognition's private coding benchmark: 132 tasks in real open-source repositories, graded on whether a maintainer would merge the agent's change, with any failed blocking check scoring zero, and each run three times in Artificial Analysis' own harness, Stirrup. Terminal-Bench-Science is 70 expert-curated research tasks run in a terminal, where a task passes only if every one of its tests passes.
The revisions are quieter. HLE-Diamond, a refined 1,000-question version of Humanity's Last Exam split evenly between reasoning and knowledge, replaces HLE outright. AA-Omniscience v1.1 strips out saturated and public-set questions and revises ambiguous ones. CritPt v2 contributes a private 40-question hard subset, run with a code-execution tool. Removed: GDPval-AA v2.1, SciCode and AA-LCR v1.1.
Because the weights are published, the swap can be priced. The three evals that left carried 25% of the v4.3 index; the two that arrived carry 30%. GDP.pdf, which the same table marks as neither private nor tool-using, was halved from 10% to 5% to help pay for them.
| Evaluation | Weight | Private test set |
|---|---|---|
| AA-Briefcase v1.1 | 15% | yes |
| FrontierCode 2.0 | 15% | yes |
| AA-Omniscience v1.1 | 15% | yes |
| Terminal-Bench-Science 0.1 | 15% | no |
| Terminal-Bench 4.0 | 10% | no |
| CritPt v2, hard subset | 10% | yes |
| HLE-Diamond | 10% | no |
| AutomationBench-AA | 5% | yes |
| GDP.pdf | 5% | no |
Those are the nine rows in the methodology table, and the five flagged private sum to exactly 60%, which reproduces the headline without rounding.
The top of the table is a max-effort table
Anthropic and OpenAI hold the top five, and no single model leads all nine evaluations: Opus 5.5 takes three, five others take the remaining six. Sonnet 5.5 tops AA-Briefcase, Kimi K3.1 Pro takes Terminal-Bench 4.0, Gemini 4 Argon takes AutomationBench-AA, Muse Wonder Full takes GDP.pdf.
The config labels in parentheses are not decoration, because effort settings matter more in v5 than in v4.3. Opus 5.5 scores 35 at low effort and 58 at max, at ten times the cost per task, $0.98 against $10.04. Sonnet 5.5 now spans 29 points from low to max, up from 20, and GPT-6.1 Sol's spread doubled from 10 to 20. A composite fed by one variant per model is reporting a ceiling, not a default.
The open-weights gap, and who counts as open
The headline is that the gap between the best open model and the leader widened from 3 points in v4.3 to 12 in v5. The top open-weights entry is Kimi K3.1 Pro at 46, and its weights have not been released. Among models whose weights are downloadable, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32, which is 26 points behind the leader and below Claude Haiku 5.5 (max) at 36.
Two caveats belong on that figure. The first is definitional: a bucket that admits an unreleased checkpoint is a bucket about licensing intent, not availability. The second is compositional. Three evaluations with comparatively procedural, publicly derived tasks left the index, and both replacements are repository and terminal work. That is where closed labs have concentrated their reinforcement-learning spend. How much of the 12 points is capability and how much is the new exam cannot be separated from outside, and the widest per-benchmark gaps show the shape of the change: Kimi scores 37% on HLE-Diamond against 68% for Opus 5.5, and 34% on Terminal-Bench-Science against 63% for Astra. We examined the earlier, narrower version of this gap in our October piece.
Sixty per cent of this cannot be reproduced
Private test sets are the standard answer to contamination, and after two years of frontier models memorising public benchmarks, the motive is sound. The cost is reproducibility. An outside lab can push a model through Stirrup, but it cannot obtain the items, so it cannot confirm that 46% on FrontierCode means what it claims. The grading is also nested judgment the whole way down: an LLM judge applies a rubric, the rubric encodes a maintainer's merge standard, and the maintainer stands in for a decision nobody actually made.
Artificial Analysis publishes one partial escape hatch. AA-Omniscience-Public-v1.1 is a public subset recalibrated so that accuracy, hallucination rate and total performance match the private set. That is a reasonable gesture, and still a claim about its own construction, checkable only in the direction the publisher chose.
Why a second index now exists
The Progress Index v1 is the most honest thing in the release. Its methodology states the problem in one sentence: "Each version of the Intelligence Index is therefore measured on a different set of evaluations, and scores across versions are not intended for long-run comparison." The Progress Index instead fits model capability and item difficulty jointly with item response theory, linking models through tasks they have in common, so a 2023 model and a 2026 model land on one scale. It is anchored at o3-mini (high) on 220 and GPT-5.5 (xhigh) on 320, and it builds on Epoch AI's Capabilities Index, which the authors credit.
It is not a clean ruler either. The model assumes a single latent capability dimension, a strong assumption for a field where coding and physics research are diverging, and each effort variant is fitted separately, with each model represented by its highest-scoring variant. That imports the max-effort problem into the long-run series too.
What a buyer actually pays for a point
Among the ten highest-scoring models shown, cost per Intelligence Index task spans more than 13 times, from $0.96 for GPT-6.1 Terra (max) to $13.04 for Claude Fable 5.1 (max). Opus 5.5 costs $10.04 per task, Astra $6.24, Sol $1.27. Sol sits within four points of the leader at roughly an eighth of the per-task cost, which is the most useful sentence in the release for anyone choosing a model rather than ranking one.
The reason is visible in the token counts. Every model measured generates more output per task than in v4.3, median up by two thirds from 35,000 to 59,000. The four agentic evaluations are 55% of the index weight and a median of roughly 90% of output tokens, and Terminal-Bench-Science alone accounts for about a third. The index did not get harder to score well on by accident; it got harder in a way that bills by the token.
The table has already moved
Gemini 4 Carbon took the top spot on October 28, a day after v5 shipped, per Artificial Analysis' own follow-up. A v5 leaderboard is a snapshot with a short shelf life, and the revision that produced it will happen again, because benchmarks that differentiate today's leaders saturate the way HLE and SciCode did.
Read the score and the cost together, check the effort setting behind it, and never difference it against a number from another index version. Those three habits cover most of what the release supports. The remainder is a well-argued case that Artificial Analysis' private exam is the fairer one, and no outside party is currently positioned to check the marking.
