AI news · October 17, 2026
Mercor maps a large gap between AI benchmarks and real work
Mercor found only 208 of 923 occupations with enough benchmark prompts and estimates 90% of US working hours lack good coverage, so benchmark scores miss much of buyer work.
- occupations with enough prompts
- 208/923
- working hours with little coverage
- 90%
- job families with no such occupations
- 6/22
Mercor published a benchmark map that appeared on Hacker News on October 16. In its dataset, only 208 of 923 occupations have a task with at least five benchmark prompts, and six of 22 job families have no such occupations. Mercor says this leaves 90% of American working hours on tasks with little or no coverage from existing benchmarks.
The result is a critique of what model scores can tell a buyer, not proof that all benchmark work is poor. The page includes model comparisons, but the bigger message is about task selection: a system can look strong on a public test while missing the messy decisions, documents, and exceptions in a real job. Teams need their own accepted-task set.
What you can do with it
List the ten tasks that consume the most time in one role and write five real examples for each. Score a model on the final accepted result, correction time, and cost, then compare that score with any public benchmark. Add failed and rejected outputs to the set instead of testing only clean cases.
Our take
This is one of the most useful numbers in the issue because it changes how a buyer should evaluate models. Public benchmarks still help, but they cannot replace a task set made from real work. The commercial advantage goes to products that can measure accepted output on the buyer’s own examples.
Links Read the benchmark map
Source: Hacker News — Made With Models writes the brief; the reporting is theirs.