The open-weight gap is 7 months, 4 months, or 14 — it depends whose ruler you count
Epoch's index puts the gap at four months, its own furniture benchmark at seven, a one-person leaderboard at seven, and a China tracker at four to fourteen.
Vlad Makarovreviewed and published
A poster on r/LocalLLaMA put the distance between China's open-weight models and America's closed frontier at seven months, and attached a chart he had built himself. The source was whatmodel.sh, an open-weight leaderboard he had launched the previous day, where every point links back to a published benchmark result. The thread drew a few hundred upvotes and over two hundred comments. Elsewhere in Epoch AI's own work the same gap reads four months on its aggregate index and roughly seven on a furniture-assembly benchmark, and a Startup Fortune write-up of Epoch's China tracker, published the same day as the post, gave a range of four to fourteen. One question, four answers, and none of them a mistake.
What the seven months actually counts
The leaderboard's selling point is provenance. Its author's summary says the dataset holds 22,113 published results across 987 benchmarks from 1,011 sources, each linked to where it was published, with 14,436 independent evaluations — papers whose authors ran the models, plus Epoch AI, the LMArena dataset, EvalPlus and the EleutherAI harness — against 7,677 scores labs reported for their own models. The live homepage on October 16 read differently: 879 models whose scores combine 566 benchmarks, updated 2026-10-14 08:02 UTC. Both totals are the project's own, and they count different things.
The scoring is not an average of percentages, because averaging says little when one model is graded on an easy quiz and the next on research maths. The site's method page lines every test up on one shared scale using item response theory, the technique exam boards use to put different papers on the same scale. A test that every model passes gains a small sharpness value and counts for less, and a test needs at least three scored runs before it counts at all.
Epoch's own two numbers disagree with each other
Epoch AI's data insight of May 29, 2026 states that since January 2026 the most capable open-weight models have lagged frontier closed models by an average of four months on the Epoch Capabilities Index, an average gap of eight points — roughly the distance between GPT-5 and GPT-5.5. The method is bootstrap resampling: an open model counts as having caught up if it beats a past frontier model in at least five percent of paired resamples. Requiring a strictly higher point estimate instead, Epoch says, grows the gap to six months.
Then the same organisation's IKEA furniture-assembly report of September 23 put open-weight models about seven months behind on a test of 60 photos from three builds. The report explains its own divergence: "Kimi K3 trailed the frontier by only 4.4 months, as measured by the Epoch Capabilities Index." The furniture gap runs wider because vision is still rare in Chinese open-weight releases — DeepSeek's V4 Pro and Flash lack image support, as does Z.ai's GLM 5.3. Same lab, two benchmarks, 4.4 months and seven.
Four to fourteen is a history, not an error bar
Startup Fortune reported on October 14 that Epoch "now clocks" China's open weights against America's closed frontier at four to fourteen months, averaging seven since 2023, down from well over a year. Epoch's own US-China tracker says the same: a mean of seven months, a minimum of four, a maximum of 14. That maximum is not spread around a current reading — it is the first Chinese model to beat GPT-4, in May 2024, fourteen months after GPT-4 shipped. No Chinese model has yet surpassed o3, released in April 2025, and the tracker's downloadable file was last updated on January 6, 2026. A range with a seven-month mean over three years describes the shape of a race, not where anyone stands now.
Startup Fortune also supplies the freight the headline cannot: it says the index is assembled from 39 benchmarks including FrontierMath, GPQA Diamond and SWE-Bench Verified, and credits Stanford's 2025 AI Index for the finding that the US lead on MMLU fell from 17.5 points at the end of 2023 to 0.3 a year later, with HumanEval going from 31.6 points to 3.7.
Two different questions wearing the same units
Epoch's index carries no nationality — it measures open-weight against closed-weight. The Reddit framing is geopolitical: Chinese open weights against the US closed frontier. Epoch notes the two gaps "closely resemble" each other, unsurprising given that nearly every leading Chinese model ships its weights and nearly every US frontier model does not. The sets still are not identical: OpenAI ships open weights — gpt-oss-120b sits in Epoch's own table at an ECI of 139.64 — and Google ships Gemma, while Alibaba's Qwen3-Max arrived as API access only.
| Claim | Whose ruler | What it measures |
|---|---|---|
| 7 months | whatmodel.sh, built by one person from 22,113 published results | Chinese open weights against the US closed frontier |
| 4 months | Epoch Capabilities Index (39 benchmarks, per Startup Fortune) | open weights against closed weights, no nationality attached |
| 7 months | Epoch furniture assembly, 60 photos, three builds | open against closed on one visual task |
| 4 to 14, mean 7 | Epoch US-China tracker | the historical range of the gap since 2023 |
A fourth number circulates too. Mozilla's September State of Open Source AI put the distance at about 4.4 months — Epoch's own figure for Kimi K3 on the general index.
An aggregator cannot outrun what it aggregates
A leaderboard assembled by one person out of other people's scores inherits every one of those scores' flaws. Roughly a third of whatmodel.sh's evidence is lab-reported, and the site marks it. The caveats it prints are the honest part: homepage entries carry flags reading "new or thinly tested: ranked on too few benchmark results", "only the maker's own numbers so far" and "RAM is estimated; full context may need more memory". The method page goes further, stating that the ± on a score "is uncertainty within this statistical model, not a guarantee of real-world performance or a validated confidence interval", and that using makers' numbers "like anyone else's" means only that "no one else has checked them yet".
None of that addresses the harder problems: contaminated test sets, benchmarks that saturate, and coverage that varies wildly between models. Epoch's index names a related bias in its limitations section: open-weight models "tend to perform worse on private benchmarks compared to closed models, plausibly because they more aggressively optimize for public benchmarks". If that holds, every public-benchmark gap, including the leaderboard's seven months, is an understatement.
What the thread's readers argued instead
The replies largely skipped the methodology. The most popular counter-argument was distillation: "Yea it takes 7 months to distile the us models," wrote u/antunes145. Others argued that parity was never the point. "If I want something fast that writes the best code, I pay for it. If I want anything else, I can run that local and it's fine," wrote u/NineThreeTilNow. u/Intrepid-Second6936 said the Qwen 3.8 27B release marked the first time "I'm not desperately craving improvements through new models, only looking forward to them while enjoying what already works amazing locally for me." A thread about a measurement problem became an argument about whether the measurement matters.
What would settle it
One fixed basket of uncontaminated benchmarks, publicly documented, run by a neutral third party under a single protocol with the same harness for both populations, and per-task cost printed beside every score. Epoch's furniture report already points that way: it documents its 60 items, its grading rules, its LLM grader and its limitations, which is what lets an outsider argue with the method rather than the conclusion. whatmodel.sh releases its data under a share-alike licence and its scoring code under MIT, so anyone who distrusts the seven months can recompute them. That transparency is real, and it is not agreement. Until somebody runs the identical exam for both populations, the honest answer to "how far behind are open weights?" remains: ask whose ruler, and ask what they put on the test.
