The Model Is No Longer the AI: What GPT-6 Astra’s 99.9% ARC-AGI-3 Score Actually Measures

A model in isolation versus the same model embedded in a wider system. Editorial image generated by the author.

When OpenAI released GPT-6 Astra, one number inevitably attracted most of the attention: 99.9 percent on ARC-AGI-3. Only a couple of months earlier, GPT-5.6 Sol had scored 7.8 percent on the semi-private evaluation. Read literally, the comparison suggests an almost absurd jump in capability.

There is an important qualification. Astra did not score 99.9 percent using ARC Prize’s Standard harness. Under that evaluation its best result was 62.7 percent. The 99.9 percent score came from a Provider Adapter that preserves Astra’s opaque reasoning state between requests and uses context compaction during longer interactions. ARC Prize explicitly treats the two harnesses as answering different evaluation questions.

The percentages themselves also need care. ARC-AGI-3 is not scored like an ordinary accuracy test. Its Relative Human Action Efficiency metric combines completion with the number of actions required relative to a human baseline. A score of 62.7 percent therefore does not mean that Astra simply solved 62.7 percent of the benchmark. The ARC-AGI-3 scoring method rewards both reaching later levels and doing so efficiently.

The interesting question is what to make of the enormous gap between Astra’s two results. Did the Provider Adapter inflate performance, leaving 62.7 percent as the only score that really counts? Or does the Standard harness suppress capabilities that belong to the system people will actually use?

Neither interpretation is quite satisfactory. The two scores measure different objects. Scientific comparison needs controlled conditions that make models comparable across providers. Practical capability depends increasingly on the model’s interaction with reasoning persistence, context management, memory and tools. As those surrounding components become more consequential, AI evaluation has to keep track of both.

What the Provider Adapter Actually Changes

The word “harness” can give the wrong impression. It sounds as though OpenAI surrounded Astra with specialised software capable of helping it solve ARC puzzles. If that were what had happened, scepticism about the 99.9 percent result would be straightforward.

ARC Prize describes something narrower. Under the Standard harness, the model receives the information required to solve the game and can preserve visible notes of its own choosing between interactions. What does not persist is the richer opaque reasoning state produced internally by the model. The Provider Adapter carries that state between requests and uses compaction to manage longer contexts, allowing Astra to reuse previous work instead of reconstructing it from whatever it chose to write into its visible notes.

ARC Prize’s verified Astra results show that the effect appears across every tested reasoning-effort setting.

GPT-6 Astra reasoning effort	Standard harness	Provider Adapter
Maximum	62.7%	98.6%
XHigh	59.3%	98.4%
High	54.8%	99.9%
Medium	38.6%	98.4%
Low	17.5%	98.0%
None	35.2%	96.7%

The gap is too large to treat as a minor implementation detail. Even with the API’s reasoning-effort setting at “none”, Astra reaches 96.7 percent through the Provider Adapter, while maximum reasoning through the Standard harness reaches 62.7 percent. The reasoning-level results themselves are somewhat noisy—“none” even beats “low” under the Standard harness—so this is not a clean experiment in the value of additional inference compute. The stable feature is the much higher Provider Adapter performance across the entire table.

A human analogy helps, provided it is not pushed too far. Imagine solving an unfamiliar strategy game while retaining the explicit notes you chose to write down but repeatedly losing access to much of the thought process that produced them. A second setup allows you to continue from the reasoning state you had already reached. The second player has not necessarily become more intelligent, but the difference in effective performance could be substantial.

There is an important limit to the analogy. ARC deliberately makes the decision about what to preserve part of the task. The Standard harness is not simply wiping Astra’s memory. ARC Prize argues that a sufficiently capable general system should be able to operate through a provider-neutral interface and manage its own visible working memory. That is a reasonable benchmark objective. It is simply a different objective from asking how well Astra performs with the context-management machinery OpenAI designed around it.

Sol Shows This Is Not an Astra-Specific Effect

GPT-5.6 Sol makes the harness effect harder to dismiss as an Astra-specific curiosity. On the public ARC-AGI-3 environments, Sol scored 13.3 percent using the Standard harness. OpenAI then enabled retained reasoning and context compaction. According to OpenAI’s account of the experiment, the score rose to 38.3 percent while output-token use fell by roughly a factor of six, without changing the model itself.

That result predates Astra’s near-saturation of the benchmark. It therefore provides a useful control case: execution and context architecture could already unlock considerably more effective performance from Sol.

The most useful comparison, however, is still missing. ARC Prize’s Sol results report a verified semi-private Standard-harness score of 7.78 percent. As of 25 September 2026, I have not found an equivalent published semi-private Sol result using the Provider Adapter configuration that produced Astra’s 99.9 percent. We therefore cannot cleanly divide the Sol-to-Astra jump into one part attributable to a better model and another attributable to the surrounding architecture.

Evaluation	GPT-5.6 Sol	GPT-6 Astra
Semi-private, Standard harness	7.78%	62.7%
Semi-private, Provider Adapter	No directly comparable published result	99.9%
Public, Standard harness	13.3%	—
Public, retained reasoning + compaction	38.3%	—

This also makes OpenAI’s launch table slightly awkward. It places Astra’s 99.9 percent beside Sol’s 7.8 percent even though Astra is using the provider-specific harness while Sol is represented by the Standard-harness result. OpenAI discloses the difference in a footnote, but the headline comparison is easier to absorb than the qualification.

The stronger case does not depend on that pairing. Under the same Standard harness, the score rises from 7.78 percent for Sol to 62.7 percent for Astra. Whatever combination of training, architecture or other internal changes produced that improvement, it is already a large generational jump. The Provider Adapter then reveals a second phenomenon: how much more capable the Astra system becomes when its reasoning state can persist in the way its provider intended.

Two Scores, Two Questions

Trying to choose between 62.7 and 99.9 percent as Astra’s one “real” score collapses two measurements into one.

The Standard harness answers something close to a scientific question: how well do different models perform through the same minimal, provider-neutral interface? That is exactly what controlled evaluation requires. If OpenAI, Anthropic and Google can each surround their models with arbitrary memory systems, prompts, tools and specialised software, a leaderboard quickly stops telling us which underlying model improved.

A previous Journal essay on why AI benchmarks become obsolete reached the same measurement problem from another direction. Once evaluation moves from isolated answers towards extended work, the model, its tools, surrounding software and operating conditions all contribute to the outcome. Measuring the complete system may be exactly what a user wants while simultaneously making causal comparison harder.

The Provider Adapter sits somewhere between those extremes. It is still a benchmark harness rather than a complete production environment, but it asks a more provider-specific question: how well does the model perform when it can use the context-management mechanisms built around it?

That question becomes increasingly relevant once AI moves from laboratory comparison into useful work. Someone asking an AI to investigate a company, write software, operate a computer or conduct research has little reason to insist that every useful capability reside inside the trained weights. If persistent reasoning, compaction, external memory or tools make a system more reliable, those components become part of the product being evaluated.

The situation resembles the difference between benchmarking a processor and benchmarking the computer built around it. Processor tests remain valuable because they isolate a component. They do not tell us everything the finished machine can accomplish.

Capability Is No Longer Just a Property of the Weights

For much of the recent history of large language models, improvements in AI capability could reasonably be approximated by improvements in the model. Better training produced a more capable network, and the more capable network produced better answers. That description is becoming incomplete.

Modern systems allocate substantial computation after the initial prompt. They reason for variable amounts of time, call tools, execute programs, retrieve information, preserve state across interactions and compress old context so that useful information can survive long tasks. Agentic systems also observe the consequences of their actions and revise what they do next. The trained model remains at the centre, but its weights no longer determine the capability of the whole system by themselves.

It is useful to distinguish two sources of progress, while recognising that the boundary between them is porous. One comes from improvements in the underlying model: better representations, reasoning and training. The other comes from improving the machinery that turns those capabilities into sustained behaviour: context management, memory, inference strategies and orchestration.

Astra appears to have moved along both dimensions. The Standard-harness comparison with Sol points to a major improvement under controlled conditions. The Provider Adapter result shows how much further the effective performance can move when the model is allowed richer reasoning persistence and context management.

The two effects are unlikely to be fully independent. A newer model may itself have been trained to exploit persistent reasoning more effectively. What looks like a “harness improvement” can therefore expose capabilities that only the newer model possesses. This is precisely why dividing intelligence neatly between the model and the surrounding system becomes difficult: practical capability emerges from their interaction.

Sol also shows why some important progress may occur without another large training run. Its public ARC-AGI-3 score nearly tripled when OpenAI changed how reasoning and context persisted, while output-token use fell. Agentic systems still lose track of goals, repeat work, bury useful discoveries under growing context and fail to carry conclusions cleanly across long tasks. Some of those failures reflect the model’s reasoning limits. Others reflect the machinery around it.

Improving that machinery can be much cheaper than training another frontier model from scratch. A model that is only modestly better at individual reasoning steps but far less likely to lose its place during an extended task could become substantially more useful without producing an equally dramatic movement on conventional static benchmarks. AI capability may therefore become less synchronised with the arrival of new model names.

Benchmarks Can Underestimate Capability Too

Most criticism of AI benchmarks focuses on the opposite danger. Training data may contain benchmark questions. Developers can optimise against prominent tests. More inference-time computation can buy higher scores at considerable cost. Once a benchmark approaches saturation, a handful of ambiguous or unrepresentative failures may determine the ranking.

Astra exposes the mirror-image problem. An evaluation can also understate practical capability if its interface removes mechanisms that are central to how the model normally operates.

That does not make the Standard harness defective. It makes its purpose narrower. It measures performance under common conditions, allowing a relatively clean comparison between models. A provider-specific evaluation measures something else: what happens when some of those artificial constraints are relaxed and the model is coupled to its native context machinery.

For AI evaluation, the gap between those results is itself information. If two models perform similarly under a common harness but one becomes dramatically more capable when integrated into its provider’s system, that difference matters to users even if it complicates the science. Conversely, a spectacular provider-specific score tells us less about model-to-model progress unless the same conditions are reproduced elsewhere.

The emerging problem is therefore not that benchmarks must choose between models and systems. We need measurements that make clear which one they are testing.

ARC-AGI-3 Is Not the Same as AGI

ARC-AGI was designed to probe forms of generalisation that conventional static benchmarks often miss, so a score close to 100 percent inevitably invites claims that AGI has arrived. ARC Prize itself is much more restrained.

ARC-AGI-3 consists of bounded, turn-based environments with deterministic, closed-ended mechanics and goals. Success requires exploring unfamiliar worlds, inferring their rules and acting efficiently, which makes the benchmark a meaningful test of adaptive reasoning. It does not reproduce the ambiguity, incomplete information, conflicting objectives and open-ended consequences of operating in the real world.

ARC Prize’s analysis of Astra nevertheless reports behaviour worth taking seriously. Astra used fewer actions than the median tested human on 96 percent of levels and developed compact symbolic representations of unfamiliar game mechanics that it used to track state and plan. ARC Prize describes the result as a major milestone while explicitly rejecting the claim that saturating ARC-AGI-3 by itself proves AGI.

That seems the right balance. We do not need the AGI label to make the Astra result significant. Under common Standard-harness conditions, Astra represents a very large improvement over Sol. Under the Provider Adapter, the same model reaches performance close to the benchmark ceiling. The first result tells us something about progress in the model. The second tells us how much the surrounding execution architecture can matter.

Calling 99.9 percent simply “Astra’s score” conceals an important methodological qualification. Dismissing it because Astra used a provider-specific harness conceals something equally important.

We will still need benchmarks that strip systems down to discover what changed. But if we want to know what artificial intelligence can actually do, increasingly we will also have to put the system back together again.