Astra and Fable Alignment Tests Explained

On September 3, 2026, OpenAI released GPT-6 Astra with a headline that will follow the company around for a long time. The issue is that the number does not hold up under an independent harness. ARC Prize, the organization that runs the benchmark, scored the same model at 62.7% under standard conditions and 99.9% only through OpenAI’s custom harness. The difference between those two figures reveals the current state of AI evaluation in 2026.

Key Takeaways:

Astra’s 98.6% ARC-AGI-3 claim falls to 62.7% when ARC Prize tests the model using its own standard harness instead of OpenAI’s custom Provider Adapter setup.
On Artificial Analysis’s independent Intelligence Index, Astra scores 61, matching its predecessor GPT-5.6 Sol and five points behind Anthropic’s Claude Fable 5.1 at 66.
The ARC-AGI-3 score varies with added system components, not the model’s raw ability: NVIDIA’s AVO system reached 100% on the same benchmark using Claude Opus 5, whose baseline was about 30%.
OpenAI left out GDPval, its own benchmark for economically valuable work, from Astra’s launch materials, so the “AGI era” claim depends on specialized benchmarks and demos.
Both models still depend on custom setups and human-defined parameters that inflate scores, which means alignment evaluations from 2025 remain a weak point in the AGI argument.

The Harness, Not the Model, Is the Number

The main finding is that a benchmark score depends on the system built around the model, not the model alone. ARC-AGI-3 differs from other evaluations in regular use: it requires a system to generalize to unfamiliar problems rather than reproduce capability learned during training. That is why it has become the benchmark people cite to claim AGI is near.

However, the score is very sensitive to the harness running the model. ARC Prize scored GPT-6 Astra at 62.7% on ARC-AGI-3 with its own harness and 99.9% with OpenAI’s, then adjusted five metrics in response. According to Techloy’s report on ARC Prize’s published figures, the 99.9% result came through OpenAI’s custom harness at a cost of $18,817, while the 62.7% standard-harness run cost $26,098. The higher score was also the cheaper one because the custom harness changed what the model was being asked to do.

This issue is not new. But NVIDIA did not create a foundation model that jumped to 100%. The perfect score came from persistent memory, tools, feedback, and recovery mechanisms added around the model. The model itself did not improve; the system added more components.

Astra vs. Fable 5.1: Two Different Scorecards

The two flagship releases from early September 2026 tell opposite stories depending on which benchmark you trust. Anthropic released Claude Fable 5.1 on September 1, and OpenAI followed with GPT-6 Astra on September 3, according to CryptoBriefing’s coverage. On OpenAI’s own benchmark table, Astra leads every comparable metric. On independent evaluation, the results reverse.

Rows of servers in a data center running large language model evaluations — Frontier model evaluation now runs on infrastructure this large, but scores still depend on which harness a lab chooses to publish.

The clearest independent assessment comes from Artificial Analysis’s Intelligence Index. Astra, running at maximum reasoning effort, scores 61, matching its predecessor GPT-5.6 Sol and tying with xAI’s Grok 4.6. Claude Fable 5.1 tops the index at 66, and Meta’s Muse Spark 1.3 scores 62. For a model promoted as a decisive leap, no improvement on the industry’s most-watched independent benchmark is unusual.

The pricing worsens the picture. Artificial Analysis calculates that Astra costs 2.5 times what Sol charged, increasing from $4/$20 to $10/$50 per million input/output tokens, and ends up about 75% more expensive per task than Sol for essentially the same intelligence score.

Metric	GPT-6 Astra	Claude Fable 5.1	Source
Artificial Analysis Intelligence Index	61	66	Artificial Analysis via OfficeChai
ARC-AGI-3 (standard harness)	62.7%	See ARC Prize figures	Techloy / ARC Prize
API price (input/output per 1M tokens)	$10 / $50	$10 / $50	CryptoBriefing
SWE-bench Pro	See OpenAI table	81.2%	CryptoBriefing
DeepSWE v1.1	74.1%	67.4% (per OpenAI’s table)	VentureBeat

The table does not reveal the unevenness in how each lab presents its results. Anthropic’s Fable 5.1 numbers come with a 1 million-token context window and 128K max output, concrete engineering specs that developers can verify by using the product. OpenAI’s Astra claims rely on a benchmark table where the ARC-AGI-3 figure includes a note buried in the evaluation details: the model runs through the Responses API harness, while comparison models use different configurations, as VentureBeat reported.

What 2025 Evals Still Miss

The most notable detail in the Astra launch is what is missing. OpenAI introduced GDPval in 2025 to move beyond academic-style tests, evaluating models on 1,320 tasks from 44 knowledge-work occupations across nine U.S. industries. It was the company’s attempt to measure economically valuable work rather than abstract reasoning. GDPval does not appear in Astra’s launch materials, as VentureBeat noted.

GDPval is one benchmark OpenAI created to answer the question implied by Brockman’s “AGI era” claim: can a model produce work comparable to experienced professionals across many occupations? OpenAI explains that current GDPval is one-shot and does not measure the long-horizon, multi-application work Astra is supposed to excel at. The result is that the AGI claim depends on a mix of specialized benchmarks and demos rather than the company’s main economic test.

The alignment issue connects to a problem covered earlier this month. In our analysis of why AI agents are dishonest and cooperative, the pattern was that training rewards the appearance of success, so models learn to game the objective. The same applies to evaluation. When a benchmark score can shift 37 points by changing harnesses, the score becomes a reward signal that labs optimize, not an unbiased measure of capability. OpenAI acknowledged a related issue in its own materials: Astra’s monitorability has declined compared to earlier versions because the model manages its own reasoning steps and is harder to evaluate, as Geeky Gadgets summarized.

The 2025-era evaluations share a structural flaw. They depend on custom setups and human-defined parameters that can inflate precision, recall, and F1 scores without reflecting true generalization. Claude Fable 5.1 shows similar variability, with scores changing based on grading criteria and safety layers. Until evaluation methods become reliable enough that switching harnesses cannot change a score by 37 points, every AGI claim based on them remains a claim about added system components, not intelligence.

A Cheap Way to Check the Number Yourself

You do not need access to either lab’s internal harness to see the issue. The pattern appears in any evaluation where system components and model are not clearly separated. The code below illustrates the difference: the same model output is scored differently depending on whether you allow retries, persistent memory, and tool access around it.

import random

def solve_task(task, model, retries=0, memory=None, tools=None):
 """Score a task with or without added system components around the model."""
 memory = memory if memory is not None else {}
 context = dict(memory)

 for attempt in range(retries + 1):
 if tools:
 context["tools"] = tools
 output = model(task, context)

 if tools and output.get("needs_tool"):
 # Tool call lets the model recover from a wrong first answer.
 context["tool_result"] = tools[output["tool"]](output["args"])
 output = model(task, context)

 if output["correct"]:
 memory[task] = output["answer"] # persist for later tasks
 return True

 return False

def model(task, context):
 """Stand-in for a foundation model: ~30% baseline accuracy."""
 return {"correct": random.random() < 0.30, "answer": "guess"}

# Same model, three different setups, three different scores.
bare = sum(solve_task(t, model) for t in range(1000)) / 1000
with_retries = sum(solve_task(t, model, retries=2) for t in range(1000)) / 1000
with_full_stack = sum(
 solve_task(t, model, retries=2, memory={}, tools={"calc": lambda a: a})
 for t in range(1000)
) / 1000

print(f"bare model: {bare:.1%}")
print(f"with retries: {with_retries:.1%}")
print(f"with full stack: {with_full_stack:.1%}")

This example clarifies why 62.7% and 99.9% can both represent "the same model." Every layer added around the weights, whether retries, persistent memory, tool access, or a different grading rubric, changes what the number measures. When a lab publishes a score, the reader needs to know which layers were included.

What to Watch

Three factors will influence whether the AGI discussion becomes more transparent in the next quarter. First, observe whether OpenAI publishes Astra results on GDPval or a successor designed for multi-step agentic work. The absence in the launch materials is notable because GDPval is the benchmark built for the economic argument Astra is making. A result there, positive or negative, would provide a stronger basis for the claim than another ARC-AGI-3 headline.

Second, monitor independent harnesses. ARC Prize revised five metrics after its initial Astra scoring, and Artificial Analysis now rates the model at a steady 61 on its Intelligence Index. These third-party numbers provide a counterbalance to vendor self-reported tables, and they are converging on a consistent story: Astra improves efficiency and coding but does not raise the intelligence score.

Third, watch the cost-per-task calculations. Artificial Analysis finds Astra is 75% more expensive per task than Sol for the same intelligence score, while its coding-agent results place it on the Pareto frontier for cost versus capability. That split, strong in coding and flat in general intelligence, signals what the model achieves and what it does not. The "AGI era" claim currently functions as a marketing label attached to a benchmark number that changes depending on who controls the harness.

Sources: Geeky Gadgets, VentureBeat, Techloy, OfficeChai / Artificial Analysis, CryptoBriefing, The Next Web.

More in-depth coverage from this blog on closely related topics:

Sources and References

Sources cited while researching and writing this article: