The open-weights bias accusation is three claims, and only one survives contact
A r/LocalLLaMA thread says Artificial Analysis rigs its index for closed labs. One charge is documented, one is not, and one is more serious than the post.
Vlad Makarovreviewed and publishedOn 9 November a post on r/LocalLLaMA asked whether Artificial Analysis is biased against open weight models, and the thread drew roughly 417 points and about 246 comments before it settled. The screenshot attached showed the Intelligence Index at v5.0.1, with Claude Fable 5.2 on 65. The charge was that the firm re-calibrates whenever a Chinese checkpoint pulls close to an American one, that the recalibration "shoves open weight models down the toilet", and that the people doing it "are paid in some way".
Three claims are wearing one coat. The index did change and open models did fall, documented to the decimal. The claim that competitive preference caused it has no evidence anywhere in the thread. The claim that matters, which the post stumbles into without naming, is that the revision took away everyone else's ability to check the result.
The revision is dated, published, and duller than the accusation
Artificial Analysis shipped Intelligence Index v5 on 27 October. Sixty per cent of the composite now rests on evaluations with private test sets or held-out answers. FrontierCode 2.0, Cognition's private coding benchmark, counts 132 tasks in real repositories and grades on whether a maintainer would merge the change. Terminal-Bench-Science adds 70 agentic research tasks. HLE-Diamond, 1,000 questions split between reasoning and knowledge, replaced Humanity's Last Exam; AA-Omniscience v1.1 and a private 40-question CritPt v2 subset came in, GDPval-AA v2.1, SciCode and AA-LCR v1.1 went out.
At the top little moved: Claude Opus 5.5 (max) 58, GPT-6 Astra (max) 57, GPT-6.1 Sol (max) 55, Gemini 4 Argon (high) 50, with Anthropic and OpenAI filling the first five places. What moved was the distance to the open-weights column, three points to twelve. We went through that swap, and the effort settings behind the scores, in our October read.
Nine models, one revision, and a table anyone can check
The per-model deltas reported by Trending Topics are the only hard evidence either side has.
| Model | Previous | v5 | Change |
|---|---|---|---|
| Kimi K3.1 Pro | 55 | 46 | -9 |
| DeepSeek V4.1 Flash | 45 | 32 | -13 |
| MiMo-V2.6-Pro | 46 | 30 | -16 |
| Muse Spark Open | 39 | 25 | -14 |
| Mistral Large 4 Preview | none | 25 | new |
| Claude Sonnet 5.5 | 56 | 54 | -2 |
| Gemini 4 Argon | 53 | 50 | -3 |
| Muse Wonder Full | 56 | 50 | -6 |
| GPT-6 Astra | 53 | 57 | +4 |
Strip the labels and the pattern the thread claims is genuinely there: every open entry lost nine to sixteen points, and no closed entry lost more than six. Two footnotes. MiMo-V2.6-Pro's prior score is unpublished, so its 46 is implied by the stated loss of 16. Kimi K3.1 Pro counts as the leading open-weights model at 46 while its weights were unreleased, a bucket about licensing intent rather than availability; Moonshot published the files on 6 November, ten days after v5 shipped, leaving the column with no downloadable leader.
The thread agrees with itself less than it thinks
Its most-upvoted comment, at 357 points, is u/lemon07r's: "Considering they always re-weigh the index every time a chinese model pulls ahead; yes... I do believe they needed reweighing cause some of the scores made no sense, putting some clearly dumber models near the same score as better models like opus, etc, but every time kimi or qwen catch up they decide that's when to reweigh the board for some reason." That is the strongest form of the case, and it is a claim about timing, not payment.
u/black__and__white puts it in two lines: "1. there is a conflict of interest 2. the gap is, nonetheless, widening, and believing otherwise is cope." Which concedes the direction of the gap and moves the argument onto cause.
Cause is where the thread stops producing anything checkable. One commenter, u/SexyAlienHotTubWater, asserts that the firm's backers include a board member at OpenAI and several people at Google, and that no other industry would trust a body owned by those it evaluates. Nobody produces a cap table or investor list, and nothing behind this piece substantiates it: a hypothesis that would matter enormously if true.
The most useful reply is the least exciting. u/DegenDataGuy, at 98 points, notes that old questions stop discriminating as models improve, that the benchmark gets harder when they rotate out, and that "since the open models tend to be less capable... they get hit the hardest".
What the accusation quietly leaves out
The detail that cuts hardest against the post is that the same v5 run put Kimi K3.1 Pro first in the entire field on Terminal-Bench 4.0, ahead of every proprietary model, at 67 per cent. An index engineered to put open weights in their place does not hand a Chinese checkpoint the top of its most expensive agentic evaluation. The revision also cost closed models points, with Gemini 4 Argon down three, Muse Wonder Full six and Claude Sonnet 5.5 two, while only GPT-6 Astra gained.
The board kept moving after the accusation was written. The Claude Fable 5.2 model page shows v5.0.1, the model at 65, ranked first of 74, at $26.54 per Intelligence Index task, 103 tokens a second and 44.4 minutes per task. A fresh leader for a closed model appeared within days of shipping, on a point release, which cuts both ways: the ranking is a live instrument, and the screenshot in the thread is not the v5 table at all.
The gaps the release does publish are large and consistent: Kimi 37 per cent on HLE-Diamond against 68 for Opus 5.5, 34 on Terminal-Bench-Science against 63 for Astra, with most open-weights models under 10 per cent.
The objection that survives contact
The defensible complaint has nothing to do with motive. Because 60 per cent of the index sits on private sets, an outsider can no longer verify a run, and the reason Artificial Analysis gives is sound: public questions end up in training data. Trending Topics drew the consequence plainly, that the results "can no longer be independently verified from the outside". The trade is explicit and one-directional. You can push a model through Stirrup, but you cannot obtain the items, see the rubric or read the judge's transcript, so the number stops being a measurement you can check and becomes an assertion whose confidence interval only its author can widen.
Two things sharpen that. The replacements are agentic repository and terminal work, where closed labs have concentrated their reinforcement-learning budgets, so the exam tilted toward their advantage at the moment it stopped being auditable. And the price rose with it: models emit a median of roughly 59,000 output tokens per task, two-thirds more than before, and among the ten highest-scoring models cost per task spans more than 13 times, from $0.96 for GPT-6.1 Terra to $13.04 for Claude Fable 5.1. A ranking that bills by the token while withholding its answer key asks for more trust than benchmarks have historically been given.
What would settle it
Three things would, none requiring bad faith. A per-question public subset with a documented sampling rule would let outsiders test whether the private and public halves rank models alike. Transcripts of a sample of graded runs, item by item, output against judge verdict, would show whether the grader is consistent. An audit of the scoring function itself, by someone who does not take the firm's money, would cover what secrecy cannot.
Until one of those exists, the accurate description is narrower than the post's. The index changed, open models fell further than closed ones, and part of that fall measures a harder exam rather than a weaker model. Motivation is unproven, probably unprovable from outside; the loss of outside verification is proven, and it deserved the headline.
