OpenAI’s July 29, 2026 report is less a story about a new model capability than about evaluation plumbing. The company says the same GPT-5.6 Sol model, at max reasoning effort, scored very differently on ARC-AGI-3 after it changed how the test harness carried state between actions.
In OpenAI’s public-set runs, the official harness produced 13.3%, while a Responses API harness with retained reasoning and compaction produced 38.3%. OpenAI also reported six times fewer output tokens with the modified setup. These are company-reported experimental results, not evidence that the model itself was updated or improved.
The harness changed what the model could carry forward
ARC-AGI-3 places an agent in unfamiliar interactive games and requires it to infer the objective and mechanics from experience. According to OpenAI, the official harness preserved past actions and short notes but discarded the model’s private reasoning after every action. As the interaction grew, a rolling truncation policy also removed the oldest history.
OpenAI rebuilt the harness with the Responses API. For GPT-5.6, passing previous_response_id lets the service retain private reasoning state across turns without exposing that reasoning to the client. The second change replaced rolling truncation with compaction, which reduces a long context while carrying forward state needed for later turns.
That distinction matters in a benchmark requiring repeated observation, hypothesis testing and revision. A system that loses the rationale behind earlier actions must spend later turns reconstructing it. A compacted history can preserve more continuity than simply deleting the oldest records. The comparison does not describe a change to model weights or training; it changes the state the harness makes available during inference.
Three percentages describe two sets and two harnesses
ARC Prize’s GPT-5.6 results page lists 13.33% on the 25-environment public set and a verified 7.78% on the semi-private set for GPT-5.6 Sol at max reasoning effort. OpenAI rounded those figures to 13.3% and 7.8% in its report.
The 7.78% verified leaderboard result came from ARC Prize’s semi-private set. The 13.33% and 38.3% figures are public-set runs using different harnesses. The 38.3% figure therefore cannot be read as a direct replacement for, or comparison with, the 7.78% leaderboard score.
The 38.3% result is OpenAI’s own public-set experiment with its retained-reasoning and compaction harness. It is not a new verified semi-private leaderboard result.
ARC-AGI-3 uses Relative Human Action Efficiency, or RHAE. The metric combines level completion with the number of environment actions taken relative to a human baseline. Internal reasoning operations are not counted as actions. OpenAI’s six-times-lower output-token figure is therefore a separate operational measurement, not another expression of the RHAE score.
Why state management can dominate a long-horizon evaluation
ARC-AGI-3 is explicitly designed around learning over time rather than producing a single answer. The benchmark’s games provide no natural-language instructions; agents must act, inspect the resulting frame, and update their understanding. Harness decisions become part of the tested system because they determine which observations and working state survive to the next step.
OpenAI attributes the higher public-set result to continuity. It says retained reasoning reduced repeated interpretation, while compaction preserved learned information over longer runs. The Responses API documentation describes previous_response_id as a continuation mechanism for preserved reasoning, and its compaction guide says compacted items carry key prior state and reasoning into a smaller context.
That explanation is plausible and consistent with the API mechanics, but the published report is still first-party evidence. It does not establish that either setting would produce the same gain on another model, benchmark or production agent. Nor does the headline result isolate how much of the difference came from retained reasoning and how much came from compaction.
What evaluation and API teams should take away
For evaluators, the practical lesson is to report the model and the runner together. A reproducible result should identify the model variant, reasoning effort, API, prompt, tool access, context limit, truncation or compaction policy, evaluation set, scoring version, token accounting and cost. “Same model” does not mean “same evaluated system” when state handling differs.
The comparison also raises a parity problem. The Decoder reported that ARC Prize co-founder François Chollet considered general-purpose API settings available to users acceptable when clearly disclosed, while acknowledging that provider-specific settings complicate equal treatment. That supports transparent harness reporting; it does not make differently configured leaderboard and public-set results interchangeable.
For API teams, retained reasoning and compaction are candidates to test in long-running agent loops, not universal defaults proven by this benchmark. Teams should run task-specific ablations and measure success, total input and output tokens, reasoning usage, latency, cost, failure recovery and data-handling requirements. OpenAI reported fewer output tokens here, but did not provide enough information to infer total production cost.
The durable conclusion is narrow: in OpenAI’s ARC-AGI-3 public-set experiment, changing state-management behavior changed the measured result. The evidence supports closer scrutiny of harness design, not a claim that GPT-5.6 Sol became more capable.
