Same Model, Same Benchmark: 54.8% or 99.9% Depending on the Harness

OpenAI's launch page gives GPT-6 Astra a 99.9% score on ARC-AGI-3. Open the ARC Prize results page, keep the model at the same high reasoning level, and you can also find 54.82%.

Nothing about the model changed. The evaluation harness did.

That gap is not a footnote. It is the story.

The number that should stop you

A benchmark score usually looks like a property of a model. We say a model "gets 85%" as if the number belongs to the weights in the same way a person's height belongs to their body.

Agent evaluations make that shortcut dangerous.

ARC Prize tested Astra across two conditions. At high reasoning effort, the Standard harness produced 54.82%. The Provider Adapter produced 99.95%. At max reasoning, the comparison was 62.71% versus 98.55%.

The adapter was not merely more accurate. Across the 167 game-and-reasoning pairs solved by both harnesses, ARC Prize reports that provider-adapter runs were about 3.66 times faster and used 49% fewer tokens.

The natural reaction is to ask which score is the honest one. That is the wrong first question. The useful question is: what system did each score measure?

What ARC-AGI-3 is testing

ARC-AGI-3 is not a static question-and-answer test. It places an agent inside unfamiliar, turn-based environments. The agent has to explore, infer the rules, identify a goal, plan, act, observe what happened, and adjust.

That last sequence matters. A static benchmark can often be described as input, model, output. An interactive benchmark looks more like this:

1. Observe the environment.
2. Decide what matters.
3. Preserve the right state.
4. Choose an action.
5. Read the result.
6. Repeat without losing the plot.

The model still matters enormously, but it is no longer working alone. Something has to manage the conversation across many steps. Something decides what history is carried forward, what is compressed, and what the next request contains.

That something is the harness.

Two harnesses, two questions

ARC Prize now reports two evaluation conditions separately.

The Standard harness uses a minimal, provider-neutral interface. The model can keep visible notes, but it is responsible for deciding what to preserve. This is the cleaner condition for comparing models under a shared interface.

The Provider Adapter harness uses context-management capabilities designed for the provider's model. For Astra, ARC Prize says this preserves opaque reasoning state between requests and uses compaction to manage longer interactions.

Those designs answer different questions.

The Standard harness asks: how well does this model perform when every provider meets the benchmark through the same minimal interface?

The Provider Adapter asks: how well does the model perform when deployed with the context machinery its maker designed for it?

The first emphasizes comparability. The second is closer to an optimized product stack. Neither question is useless, and neither number should be shown without its condition.

Why memory becomes capability

Imagine solving an unfamiliar game while someone periodically removes pages from your notebook.

You may still be the same problem solver, but your effective ability changes. If you forget that a symbol opens a door, or that an action failed three turns ago, you must rediscover it. Exploration gets more expensive. Plans become inconsistent. A long task turns into a collection of short tasks with partial amnesia.

That is why context management is not merely an efficiency feature in agentic systems. It can be part of the capability.

ARC Prize observed Astra building compact symbolic descriptions of unfamiliar environments. In a more tool-rich PRO-LONG setup, it created parsers, state models, planners, and game-specific utilities. The common pattern is not "the model knew the answer." It is that the system converted experience into a reusable representation.

A harness that preserves that representation lets the agent accumulate progress. A harness that forces the model to reconstruct it makes the same underlying reasoning less useful.

This is familiar outside benchmarks. A customer-support agent with no durable case state repeats questions. A coding agent that loses its plan reopens files it already inspected. A research agent without source memory contradicts an earlier finding. In each case, the model may be capable, but the system cannot hold on to the capability long enough to finish the job.

The benchmark is measuring a stack

Once tools and multiple steps enter the loop, "model performance" is shorthand for several layers:

• the model and reasoning setting;
• the system and developer instructions;
• the context-management policy;
• persistent and working memory;
• tool definitions and tool results;
• retry, stopping, and error-recovery logic;
• token, time, and cost limits;
• the evaluator's action interface.

Change one layer and the score may move even when the model ID stays fixed.

This does not make agent benchmarks meaningless. It makes their unit of analysis different. A benchmark can measure a bare model under a standardized interface, or a deployed agent system under an optimized stack. Problems begin when a result from the second category is presented as though it came from the first.

OpenAI's launch page highlights 99.9% and notes that the run used its Responses API harness with two changed settings. ARC Prize's leaderboard now exposes both conditions. That labeling is the right direction because the reader can see the tradeoff instead of receiving a single context-free number.

What practitioners should change

If you evaluate agents at work, the lesson is not to copy OpenAI's harness. It is to stop treating your harness as invisible.

Freeze the whole evaluation stack. Record the model version, reasoning effort, prompts, tools, memory policy, compaction behavior, token limits, retry policy, and harness commit. "We tested model X" is not enough to reproduce an agent result.

Report more than success rate. Include cost, elapsed time, number of actions, token use, and failure type. ARC Prize's result is especially interesting because the higher-scoring adapter also used fewer tokens across matched solved pairs. A system can be more capable and more efficient when better state prevents repeated work.

Separate standardized and optimized runs. A provider-neutral condition helps comparison. An optimized condition tells you what a production stack can do. Publishing both is more informative than pretending one number serves both purposes.

Test memory explicitly. Run ablations with and without persisted reasoning state, summaries, retrieval, and compaction. If performance collapses, you have learned that context management is a core dependency, not housekeeping.

Evaluate the system you will deploy. If your production agent has a database, a browser, and a recovery loop, a bare-model test will not predict its behavior. But if you want to compare vendors, letting every vendor bring a different stack may hide where the advantage came from. Name the question before choosing the setup.

What would make me wrong

The strongest objection is that the Provider Adapter is simply the realistic evaluation. If Astra is designed to preserve reasoning state and use compaction, removing those features may be like benchmarking a database with its cache disabled. On that view, 99.9% is the better estimate of what users can obtain from the deployed system.

That objection is partly right. Standardization can understate real product capability.

The opposite objection is that provider-specific context management weakens fair comparison. If one system arrives with opaque state preservation and another is limited to visible notes, the benchmark is no longer isolating the model. That is also right.

The resolution is not to crown one harness. It is to label both.

There are additional limits. ARC-AGI-3 uses bounded environments with deterministic mechanics and goals. ARC Prize explicitly says saturation is not proof of AGI, and the benchmark does not reproduce the open-ended complexity of the real world. The dramatic harness effect may also be unusually large for long interactive tasks where state accumulation is central. A short classification benchmark would not behave the same way.

Finally, these are reported evaluation results, not an independent audit performed here. ARC Prize publishes its policy, configurations, scores, and replays, which makes scrutiny possible, but practitioners should still reproduce the pattern on their own tasks before generalizing it to production.

The practical conclusion

The 54.82% and 99.95% results are not competing answers to one clean question. They are answers to two different questions that happen to share a model name and benchmark name.

That is the real benchmark lesson.

As AI systems become more agentic, the wrapper stops being a wrapper. Memory changes what can be learned. Compaction changes what can be retained. Tools change what can be attempted. Control logic changes whether a capable model reaches the end.

The model did not change. The system around it did. The score moved by 45 percentage points.

Your evaluation report should make that system visible.

Key takeaways

1. At high reasoning effort, Astra scored 54.82% with ARC Prize's Standard harness and 99.95% with the Provider Adapter.
2. The Standard harness supports provider-neutral comparison; the adapter measures the model with provider-designed context management.
3. Agent benchmarks evaluate a stack, not just model weights.
4. Memory and compaction can affect capability as well as cost and latency.
5. Report standardized and optimized results separately, with the complete evaluation configuration.
6. ARC-AGI-3 saturation is a major result, not proof of AGI or general real-world reliability.

Sources

• GPT-6 Astra verified results, ARC Prize, September 2, 2026.
• OpenAI's GPT-6 Astra on ARC-AGI-3, ARC Prize, September 2026.
• ARC Prize Verified Testing Policy, ARC Prize.
• GPT-6 Astra: A new generation of intelligence, OpenAI, September 3, 2026.