Mercor mapped AI benchmarks onto 18,797 job tasks — and found a gap it also sells the fix for

Mercor published a Benchmark Map of the US economy: an index of 29 AI benchmarks mapped onto the US Department of Labor's O*NET occupation and task taxonomy. Its headline finding is that roughly 90% of American working hours are spent on tasks with little or no coverage from existing benchmarks. The write-up reached the front page of Hacker News on 16 October, and the number is already circulating as proof that AI evaluation measures the wrong work. The direction is plausible. The precision is not. Mercor is also not a neutral observer: it sells expert data and evaluation services to AI labs, its own APEX benchmarks sit inside the index it is measuring, and the gap it documents is the market it is selling into.

What the map counts, and how

O*NET divides the US workforce into 923 occupations and 18,797 tasks. Mercor picked 29 benchmarks for what it calls quality and economic relevance, requiring at least 25 publicly available tasks, agentic rather than advisory tasks, economic realism, a cap on how many benchmarks any single job family could contribute, and benchmarks frontier models had not yet saturated. Following work by Wang et al., an LLM judge tags each benchmark prompt with as many O*NET tasks as it thinks correspond, a method validated against human expert tags on Mercor's own APEX benchmarks. A task is then scored as covered once the judge has mapped at least five prompts to it. The 90% figure layers one more input on top: Hatgis-Kessell et al.'s LLM-derived estimates of how much working time each task consumes.

Under that rule, 208 of 923 occupations, about 22.5%, have at least one task with five or more mapped prompts, and 924 of 18,797 tasks qualify. Six of the 22 job families have no occupation clearing the bar at all.

Where the coverage lands

Job family	Occupations covered
Computer & Math	29 / 36
Business & Financial	33 / 48
Legal	5 / 7
Architecture & Eng	19 / 56
Management	21 / 56
Healthcare Practitioners	12 / 89
Education & Library	9 / 62
Production	5 / 107
Transportation	1 / 52
Personal Care	0 / 31

The spread is the useful part of the map, and it is not subtle: computer and mathematical occupations sit at the top, production, transportation and personal care near the bottom. Read as a description of where the measurement industry has pointed its instruments, that ranking is hard to dispute.

Coverage is a count, not a score

A task counts as covered because at least five prompts exist somewhere in the 29 benchmarks that a judge said correspond to it. That says nothing about whether any model is accurate on the task, how hard it is, or whether it has been saturated for a year. The map measures the breadth of the instruments, not the quality of the readings. "90% of working hours are uncovered" is therefore a claim about the evaluation industry's blind spots, not about what AI can or cannot do, a distinction one Hacker News commenter put plainly: the figure mainly says there are far more different tasks than there are benchmarks.

The company holding the ruler

Mercor's business is selling expert data and evaluation to AI labs, and APEX, its own benchmark family, is one of the 29 in the index. That does not make the finding wrong, but it changes what kind of document this is. A company that sells measurement has published a finding that measurement is inadequate and needs to be expanded, and selected the benchmarks, including its own, that define the baseline. The honest reading is a vendor's argument with evidence attached, not a neutral audit. The evidence deserves to be weighed on its merits; the framing should not be inherited.

The five-prompt line

Five is a choice, and the headline rests on it. Raise the bar to ten mapped prompts and fewer tasks are covered; drop it to three and more are. Because the coverage flag feeds the working-hours aggregate, the threshold propagates straight into the 90%. What survives a different threshold is the ordering: which job families sit near the top and which near zero is stable in the map as published. What does not survive is the single number. "90% of working hours" is one point estimate built on one cut-off and one external estimate of task time shares, not a measured quantity. There is a second circularity worth naming. The tagging is done by a language model judging what other benchmarks contain, the same class of tool whose limits the map is complaining about.

What the wage-bill chart does and does not show

The map's second result is that coverage tracks occupation wage bills. Controlling for whether a task carries an AI-exposure tag from Eloundou et al., tasks in occupations with larger total wage bills are significantly more likely to be covered. The plot on the page is a task-level logistic regression with pointwise 95% delta-method confidence intervals and standard errors clustered at the occupation-group level — wide bands, an association, not a mechanism. The illustrative example is concrete and checks out: benchmarks in the index heavily cover the tasks of AI-exposed lawyers while barely covering the O*NET tasks of similarly AI-exposed legal secretaries or court clerks, leaving the long tail of smaller, middle-income knowledge work undercovered. The page also reports that AI-exposed tasks are about six times more likely to appear than unexposed ones, and that the tasks O*NET experts rate most important are six to ten times more likely than the least important. None of this shows labs deliberately skipping low-wage work. It shows where the instruments went, which is why the labour-market forecasts that numbers like these get folded into deserve their own scrutiny.

What Mercor admits

The limitations section is the most trustworthy part of the document, and it belongs in any summary of it. The authors call the results provisional and say an incomplete benchmark list necessarily biases coverage estimates down by an amount they cannot bound. They note that capping benchmarks per job family, done to widen breadth, biases coverage down in the most-benchmarked families such as software engineering. They concede that excluding saturated benchmarks risks understating coverage on easier tasks, and that they have no way to weight benchmark items by quality: some are saturated, some simplistically graded, some reward-hackable. They note that existing evaluations mostly measure automation while real use is augmentation, and that less than a third of the O*NET tasks relevant to the four occupations they scored had enough benchmark coverage to be included at all. Finally they cite Dingel and Neiman, whose estimate that only 37% of US jobs could be done entirely from home implies that about 63% involve some physical-world component today's benchmarks miss entirely.

What would settle it

Two things, neither of which the map provides. First, a per-task accuracy number for the benchmark items that already exist. The map can say a task is covered; it cannot yet say how well any model does on it, because Mercor states it has no way to weight by quality — a gap that even benchmarks built and graded by their own authors do not close. Second, independent replication of the O*NET tagging by someone with no product to sell, and a sensitivity analysis over the five-prompt threshold showing how far the 90% moves. The open-weight gap debate is a useful precedent for how badly a single headline number travels once different institutions measure the same thing with different rulers. Until then the Benchmark Map is a useful account of where the benchmark industry has looked, drawn by an interested party, with the strongest caveats disclosed in the back matter where few readers of the headline will go.