GPT-6.1 Sol found errors in nearly 800 arXiv math papers, and the authors' own caveats matter more than the headline

An ETH Zurich team ran GPT-6.1 Sol across arXiv mathematics preprints and reports that the model finds major mistakes in 6% of them and minor ones in another 30%. The result travelled fast: an r/singularity thread and a post by @haider1 with a couple of thousand likes compressed it to "major errors in 6% and minor mistakes in another 30%," plus a line about AI cleaning up human slop. The authors' own account is more careful in three places. Around one in ten of the reported major errors is probably not real, they cannot rule out that the fall in error rates has nothing to do with AI, and they have decided not to tell any of the affected authors.

What the pipeline is actually measuring

The authors explain why they did not simply ask a model to find mistakes in a paper. Agents overestimate what they find, so a small typo can be labelled major; judging whether a flaw in a proof destroys the result is extremely hard; and with real error rates low, a modest false-positive rate distorts the picture. So they narrowed the task to something checkable: find a counterexample to one of a paper's main theorems. A counterexample is easier to state and verify than a proof gap, and the prompt is blunt that a proof gap alone never counts. Unlike formalisation work such as the machine-checked results around the Erdős problems, nothing here examines the steps of a proof.

The four stages are theorem extraction, counterexample search, verification, and a follow-up-version check. The extractor takes at most five main theorems per paper from the raw TeX of version one, each with the assumptions needed to read it, and may not consult later versions. The search stage then hunts for objects that satisfy the hypotheses and break the conclusion, filing each under a severity label: edge, where the authors clearly meant something else and a typo or a missing common assumption makes the statement false as written; minor, where the statement is mathematically false in some cases but a small correction leaves the main contribution intact; or major, where salvaging the result needs substantial weakening, major reworking, or abandoning a central claim. The prompt reserves that top label for those cases, and a statement failing only in peripheral edge cases does not qualify. Everything ran as a Codex agent with GPT-6.1 Sol at high reasoning effort, for a reported cost of around $2,500 in API credits.

Two rows, and the arithmetic behind "nearly 800"

The headline split compares two windows.

Period	None	Edge	Minor	Major
Jan 2025 – Jul 2026, 656 papers	35.67%	27.90%	30.49%	5.95%
Aug – Sep 2026, 137 papers	60.58%	21.90%	16.06%	1.46%

The counting unit is a paper, and the two windows add to 793, which is where "nearly 800" comes from — a rounding rather than a round number. At the level of theorems the picture is starker: 2.28% of 2,543 extracted theorems carried a reported major error before August, against 0.54% of 556 in the later window. The final stage checked later versions of the same papers and reports that where a revision exists it lowers the severity of major errors in half of cases and cuts the frequency of any error by 28%.

Where the viral version and the paper part ways

The verification stage is the first thing the summary numbers hide: re-examining the same counterexamples changed the severity of only 2% of findings, so that step barely corrects anything. The bigger caveat is the false-positive rate. To estimate it, the authors hand-checked 20 errors, then reran the search stage with Opus 5.5 on the 75 theorems whose major errors had survived, and sampled 75 theorems from each other category as a baseline. Opus was more lenient overall and agreed with GPT-6.1 Sol on roughly 80% of the major cases; after reviewing the disagreements manually, the authors put the correct share of Sol's major findings at a bit under 90%, which the post rounds to around 10% false positives.

The second caveat is the explanation. Between August and September 2026 the reported major-error rate falls to 1.5%, and the paper ties that to the rise of AI use, citing Epoch AI's tracker, whose graph shows AI acknowledgments in over 25% of math papers in the period. The authors then write that they "cannot exclude the possibility that confounding factors affect this proportion," and supply the sharpest example themselves: GPT-6.1 Sol might be less able to identify mistakes in papers that were written with its help. Papers that acknowledge AI show the same movement as the rest, from 5.4% to 1.8%, which undercuts the idea that the effect is only about who admits using AI without settling the question. The AI-share figure the argument leans on is itself model-produced: Epoch detects acknowledgments with GPT-5.6 Sol and validates them with Claude Fable 5.

One mathematician has already challenged the direction of the numbers. Daniel J. Bernstein replied to the author's thread on X that the percentages are lower bounds on the error rate, to the extent the counterexamples are correct, not upper bounds, so the word "only" is not justified. That reading is right: the pipeline surfaces errors a counterexample can demonstrate, and silence from it is not evidence of correctness.

Why nobody is being told

The most unusual decision in the release is what the authors do with the findings: nothing. They will not actively contact authors whose papers likely contain mistakes. Two reasons are given. The first is the error rate above. The second is subtler: forcing people to verify AI-written text is counterproductive, and, in their words, "when a mistake is found, one has to acknowledge that an autonomous system quickly found issues in a work that might have taken months or years to produce." Their suggestion instead is that mathematicians run agents over their own back catalogue, keeping discovery and correction with the person who did the work. All the raw data and every stage's prompt are published, which is what makes an outside check possible.

What would settle it

The study is reproducible in principle, and its authors are candid that the numbers are estimates. What would move the picture: an independent group rerunning the pipeline and publishing its own severity calls; a hand audit of a random sample of flagged papers by specialists in those subfields, the only way to pin the false-positive rate rather than bracket it; and a measure of AI use not built by a model. There is a recent precedent for how much work that is: a fifty-problem physics benchmark whose own authors wrote, ran and graded the tests. Until then the defensible claim is narrow. On one defined class of mistake — a main theorem that a counterexample can falsify — a single model, under one set of prompts, reports a real problem in a sizeable minority of preprints, and reports fewer of them in the newest ones for reasons the authors cannot separate from the tool getting worse at the job.