When AI Passes the Test, the Test Fails: Why AI Benchmarks Become Obsolete
On 23 February 2026, OpenAI announced that it would stop reporting results from SWE-bench Verified, one of the most widely used tests of AI coding ability. The benchmark had once provided a useful signal of progress. By then, according to OpenAI’s own audit , its remaining failures had become difficult to interpret. The company examined 138 problems that OpenAI o3 had failed to solve consistently across 64 independent runs and concluded that 59.4 per cent contained material issues in their tests or descriptions. Some rejected functionally correct solutions because they demanded a particular implementation. Others tested requirements the task had never stated. OpenAI also found evidence that leading models had encountered at least some benchmark problems or solutions during training. This does not prove that SWE-bench Verified had become useless, nor should a model developer’s audit of a benchmark be treated as neutral arbitration. It reveals a recurring problem in AI evaluation: a ...