When AI Passes the Test, the Test Fails: Why AI Benchmarks Become Obsolete
On 23 February 2026, OpenAI announced that it would stop reporting results from SWE-bench Verified, one of the most widely used tests of AI coding ability. The benchmark had once provided a useful signal of progress. By then, according to OpenAI’s own audit, its remaining failures had become difficult to interpret.
The company examined 138 problems that OpenAI o3 had failed to solve consistently across 64 independent runs and concluded that 59.4 per cent contained material issues in their tests or descriptions. Some rejected functionally correct solutions because they demanded a particular implementation. Others tested requirements the task had never stated. OpenAI also found evidence that leading models had encountered at least some benchmark problems or solutions during training.
This does not prove that SWE-bench Verified had become useless, nor should a model developer’s audit of a benchmark be treated as neutral arbitration. It reveals a recurring problem in AI evaluation: a test can continue producing exact percentages after those percentages have stopped producing exact distinctions.
The frontier can keep moving after the ruler has run out of marks.
A Test That Ran Out of Room
Benchmarks make an opaque technology legible. Every model receives the same tasks, the answers are graded under common rules, and performance becomes a number that researchers, companies and users can compare. Without that common ground, model launches would depend even more heavily on demonstrations selected by the companies releasing them.
Yet a benchmark can serve several purposes, and they should not be confused. It can establish that a model has reached a minimum level of competence. It can distinguish one frontier system from another. It can also attempt to predict how well performance will transfer to work outside the test. A benchmark may retain the first function after losing the second, and it may never have performed the third particularly well.
Suppose several leading models answer more than 95 per cent of a thousand questions correctly. Almost all the benchmark’s power to rank them now rests on the remaining few dozen items. An ambiguous answer key, an unusually sensitive prompt or a handful of defective questions can reorder the leaderboard without revealing any important difference in capability.
The score remains numerically precise. The measurement does not.
This is why benchmark saturation matters. The problem is not merely that high scores leave little room above them. Most of the test is now spent confirming abilities the strongest models already share. The ranking depends increasingly on the small and possibly unrepresentative residue they still get wrong.
A saturated benchmark may remain valuable for smaller models or as a threshold test. Using it to make fine distinctions at the frontier is another matter. At that point, passing the examination has become an entry requirement rather than a useful ranking device.
When the Test Enters the Training Loop
A benchmark is most informative when it samples a capability developed independently of the test. Once the benchmark becomes influential, that independence begins to erode.
Developers study where their models fail. They generate training material resembling difficult benchmark tasks, tune prompts around the grading procedure and build tools or reasoning methods that improve the reported score. Even without direct exposure to the original questions, a model can become adapted to the benchmark’s format, subject distribution and preferred style of solution.
The exam has entered the curriculum.
That is not automatically evidence of gaming. Training students on algebra problems resembling an algebra examination may simply teach them algebra. Optimising models for coding tasks can produce genuine improvements in coding. The decisive question is whether those gains transfer beyond the benchmark’s particular structure.
If a coding test rewards short fixes to repositories with clearly described defects, better performance may indicate useful software-engineering ability. It may say much less about handling contradictory requirements, incomplete documentation, architectural trade-offs or a project in which no one has supplied a clean statement of the problem.
Direct contamination creates the clearest distortion. A model that has encountered the task or its solution possesses information unavailable to a genuinely unexposed system. Narrow adaptation matters even when no question has leaked, however. The benchmark can become less independent simply because the development process has reorganised itself around what the benchmark rewards.
This is the familiar logic of Goodhart’s law, but it should not be reduced to the slogan that every target becomes worthless. Metrics do not become meaningless the moment people optimise for them. They become harder to interpret. A rising score may reflect transferable improvement, test-specific adaptation, greater inference expenditure or some mixture of all three. The number alone cannot tell us which.
Why Harder Benchmarks Only Buy Time
The straightforward response to saturation is to construct a more difficult examination. When popular academic benchmarks such as MMLU began yielding scores above 90 per cent from leading models, researchers created Humanity’s Last Exam, a collection of 2,500 specialist questions intended to probe the frontier of expert knowledge.
This restores separation. A test on which the best systems score modestly is more useful for ranking than one on which almost every leading model approaches perfection. It does not solve the underlying problem.
Difficulty and relevance are not the same property. A question can be exceptionally difficult because it requires obscure terminology, specialised knowledge or an unusual mathematical technique. That may reveal something worth measuring. It may reveal little about whether the model can complete an ordinary but extended piece of work without losing track of its objective.
A model capable of answering an expert scientific question may still mishandle a routine administrative process because one condition was buried in a long document. Another may perform less impressively on academic material while proving far more dependable at document analysis, classification or software maintenance.
Harder static benchmarks also inherit the life cycle of the tests they replace. They begin as unfamiliar challenges. Developers study them, generate similar training tasks and raise performance. The new benchmark gradually becomes part of the environment it was meant to observe.
One response is to replace a fixed dataset with a continuously changing test. LiveBench, for example, draws frequently updated questions from recent competitions, papers, datasets and other new material. Its answers are designed to be objectively verifiable, reducing both contamination and dependence on subjective AI judges.
Dynamic benchmarks are promising, but they exchange one set of difficulties for another. Updating questions requires continuing labour and validation. Automatically generated tasks may contain hidden errors. Keeping part of a test private reduces direct optimisation but also reduces transparency: users must trust the organisation operating the benchmark without being able to inspect every question.
There is no final examination waiting to solve the measurement problem. Harder and more frequently refreshed tests can, at best, prevent the instruments from becoming obsolete quite so quickly.
From Leaderboards to Capability Profiles
Public model comparison often implies that intelligence can be arranged along one scale. Model A scores slightly higher than Model B and is therefore presented as the more capable system.
That conclusion becomes unstable once the operating conditions are examined. One model may be stronger at abstract reasoning but more prone to invented citations. Another may be more factually reliable but weaker at difficult mathematics. A third may achieve similar results at a fraction of the cost. A fourth may take the lead only when allowed to generate thousands of reasoning tokens or attempt each problem several times.
Which model is best? The question is incomplete unless it specifies the task, the acceptable failure rate and the available budget.
Evaluation researchers have recognised this for years. Stanford’s Holistic Evaluation of Language Models, or HELM, was built to examine not only accuracy but also properties such as calibration, robustness and efficiency. The field has not suddenly discovered that model capability is multidimensional. The stronger criticism is that public leaderboards and product marketing still tend to compress those dimensions into a single implied hierarchy.
Reliability is particularly resistant to compression. Two systems might both succeed in 99 out of 100 cases. The first fails by producing an incomplete answer and signalling uncertainty. The second produces a detailed, confident fabrication. Their average accuracy is identical; their practical risk is not.
The same applies to consistency. A model that occasionally produces an exceptional solution and fails unpredictably on other attempts may receive the same average score as a system that performs slightly less impressively but much more reliably. For many real tasks, the second system is preferable.
Efficiency adds another axis. A result achieved through multiple attempts, extensive tool use and a large inference budget should not be compared without qualification to one produced quickly and cheaply. Additional computation is itself a capability: it can make a system more useful when accuracy justifies the expense. But cost and latency belong in the result, not in a footnote.
The future of evaluation is therefore likely to look less like a school report and more like a technical specification. Accuracy will remain important, but alongside calibration, robustness, failure severity, consistency, speed, cost, tool use and performance under changing conditions.
The messier answer may simply be the honest one.
Evaluating Work Rather Than Answers
For systems expected to act rather than merely answer, the submitted response is only the first checkpoint.
Generated code may look plausible, but the relevant questions are whether it passes independent tests, integrates with the existing system and avoids introducing new defects. A research report matters only if its claims survive verification. An agent has not completed a task because it produced a confident description of what completion would look like.
Longer tasks also expose a weakness hidden by isolated questions: errors compound. If a hypothetical process required one hundred independent steps, each completed correctly 99 per cent of the time, the probability of finishing the entire chain without an error would be only about 37 per cent. Real agent failures are not independent, so the calculation is illustrative rather than predictive. An early mistake may be corrected later—or may distort every decision that follows.
Short benchmarks usually reset the environment after every question. Real work does not. The model must retain the objective, notice when an assumption has failed, interpret feedback and recover without quietly replacing the task with an easier one.
METR’s work on AI task-completion time horizons estimates the duration of serial human work that a model can replace at a specified probability of success. This captures something a static question set misses: the ability to sustain useful action across increasingly extended tasks.
It is not a clean replacement for ordinary benchmarks. METR has stressed that its estimates carry wide uncertainty, differ sharply between domains and do not mean that every task below the reported duration can safely be delegated. Its current tasks are concentrated in software and research, and the suite itself begins to lose resolution as models improve.
That limitation is instructive. A more sophisticated measure does not escape the instrument problem; it encounters a new version of it.
Outcome-based evaluation also forces evaluators to decide what counts as success. A software patch may pass the available tests while being difficult to maintain. A report may produce a useful decision despite containing minor errors. A fast result may be preferable to a marginally better one delivered too late.
Once evaluation moves from answers to consequences, value judgements enter the test more visibly. The model, its tools, the surrounding software, the prompt and the human supervisor all contribute to the outcome. Measuring the complete system may be exactly what users need, but it can obscure which component actually improved.
Who Judges the Judge?
Open-ended outputs create another difficulty: someone must decide whether they are good.
Human experts are expensive, slow and inconsistent. They become tired, disagree about standards and may prefer familiar approaches. Automated grading is cheaper when the answer has a verifiable form, but essays, plans, explanations and design decisions rarely have one correct string.
AI models are therefore increasingly used to evaluate the outputs of other AI models. This can scale evaluation and apply a detailed rubric consistently. It also turns the judge into another uncertain instrument.
Research on position bias in LLM judges has found that changing the order of two candidate answers can alter which one a judge prefers, with the effect depending on the judge, the task and the quality gap between the answers. Other evaluation methods can reward verbosity, familiar phrasing or styles resembling the judge model’s own output.
This does not make AI judges useless. It means their behaviour must be calibrated rather than assumed. Candidate answers can be presented in both orders. Several judges can be compared. A sample can be checked against expert ratings. Results can report disagreement instead of hiding it inside a final average.
The demand for a single ranking encourages the opposite. Uncertainty is removed, operating conditions disappear and small numerical differences become positions on a leaderboard. A model moves from third to first, while the reader is told little about whether the gap exceeds the noise of the evaluation itself.
The Instrument That Changes Its Subject
Better AI evaluation will require more than a replacement benchmark. It will require an evaluation system.
Such a system would combine public tests with private or delayed-release tasks, refresh questions regularly, repeat trials, control inference budgets and report confidence intervals. It would distinguish academic reasoning from factual reliability, extended task completion, tool use and cost. Independent organisations would test claims that model developers currently make about their own systems.
This approach will be less convenient than a universal leaderboard. It will also create new opportunities for selective reporting. With enough dimensions, every provider can find a metric on which its model appears superior. Standardised disclosure will matter as much as standardised testing: which model version was used, which prompt, how many attempts, what tools were available, who graded the result and how much computation was consumed.
There may no longer be one defensible answer to the question of which model is best. There may be only better answers to narrower questions: best at what, under which conditions, at what cost and with what kind of failure?
Scientific instruments are often described as passive extensions of perception. A telescope does not alter the star whose light it gathers. An AI benchmark is different. Once its score becomes valuable, it begins to influence training data, research priorities, product decisions and the claims made on behalf of the systems it measures.
The test does not simply become obsolete when the models learn to pass it. It may have helped build the systems that made it obsolete.
Comments
Post a Comment