Cognition SWE-2 Review: Questions and Answers

• Cognition announced SWE-2 on September 10, 2026, post-trained from Moonshot AI’s Kimi K3, a 2.8-trillion-parameter model already RL-trained for agentic coding.
• The 50.0% FrontierCode 1.1 Main score is vendor-reported on Cognition’s own benchmark, within one point of Fable 5.1 (50.9%) at a claimed 64% lower cost.
• Versus SWE-1.7, Cognition reports 58% fewer turns, 81% lower average cost, and first real code edit after a median of 18 steps instead of 48.
• A single RL run produces all three effort levels (medium, high, max) by applying a linear cost penalty tuned to the base model’s Pareto frontier slope.
• SWE-2 is distributed through Devin across Desktop, CLI, Web, and Fusion. No open weights and no published per-token price.
• Terminal-Bench 4 is the clearest published weakness: 27.3% for SWE-2 against 55.8% for Fable 5.1 and 57.9% for GPT-6 Astra.

What Cognition Shipped on September 10, 2026

On September 10, 2026, Cognition published a number that seemed unlikely for a company its size: 50.0% on FrontierCode 1.1 Main. Anthropic’s Fable 5.1 scores 50.9%. Cognition says its new model, SWE-2, reaches that score at 64% lower cost.

Two years ago that gap would have been a rounding error favoring the frontier lab. Now it is one point. The base model is not Cognition’s. SWE-2 is post-trained from Moonshot AI’s Kimi K3, a 2.8-trillion-parameter model that had already undergone extensive reinforcement learning for agentic coding before Cognition worked on it. Cognition says its own RL “still finds substantial headroom,” adding 5-6 points on many benchmarks. That continues the pattern from SWE-1.7, which was trained from a Kimi K2.7 base.

This time, the training involves a single RL run that creates three reasoning-effort levels: medium, high, and max. Cognition applies a linear cost penalty per effort level, tuning each penalty to the local slope of the base model’s Pareto frontier. That geometry matters. If the penalty is set too high, the model is rewarded for doing less work rather than better work, causing the entire cost-performance curve to shift instead of just one point moving.

Cognition also describes infrastructure improvements behind the run: a prefill delayer that batches nearby requests in the GPU scheduler for a 10-20% increase in tokens per GPU and tokens per second per request, speculative decoding through a draft model retrained online to keep tracking the policy, and NVFP4/FP8 kernels with quantization-aware training. It tripled the number of RL environments and established a process where earlier SWE-2 checkpoints generate rollouts that strengthen verifiers.

FrontierCode Is Cognition’s Own Benchmark

The 50.0% figure comes from a benchmark Cognition developed and operates. FrontierCode measures mergeability: whether a repo maintainer would actually merge the pull request the agent wrote. According to the FrontierCode leaderboard page, more than 20 open-source developers wrote tasks and spent over 40 hours per task defining what “mergeable” means in their repo. Grading combines unit tests, rubrics, and verifiers, and every task is manually reviewed by a Cognition researcher.

That is a stricter standard than pass-the-unit-test scoring, and it is the right target for agentic coding. It is also a self-written metric that no third party has run. The 50.0%, the claimed 64% cost advantage over Fable 5.1, and the “quarter of GPT-6 Astra’s cost” framing all come from the company that built both the model and the benchmark. Treat them as vendor-reported until someone reproduces them independently.

FrontierCode 1.1 was a revision published specifically to tighten internet-use rules. Cognition’s 1.1 methodology post says it added a fair-internet-use prompt plus a verifier that zeroes runs which open the upstream pull request diff, and that it demoted 75 overly strict blocker criteria. Both changes affect scores, so 1.1 numbers are not comparable to 1.0 numbers. Cognition reports that unfair-internet-use rates fell below 1% for every model it evaluated once the prompt was in place. That is a vendor self-report about the vendor’s own controls, but the control is at least concrete and auditable.

Efficiency Gains in Turns, Not Just Price

The more concrete claims are behavioral, and they rest on a comparison against Cognition’s own prior model rather than a competitor. Cognition reports that SWE-2 medium scores higher than SWE-1.7 on FrontierCode 1.1 Main while taking 58% fewer turns and costing 81% less on average. It also reports that the model makes its first real code edit after a median of 18 steps, compared with 48 for SWE-1.7.

The step-count claim is worth monitoring. SWE-1.7 drew user complaints for over-exploring simple tasks, and the fix was not a smaller model but better judgment about which parts of the codebase actually matter. Fewer turns mean less wall-clock time and fewer tokens spent on redundant reads, which lowers the cost-per-task number. Cognition’s own figures show mean steps per run falling from 127 for SWE-1.7 to 53 for SWE-2 medium, rising again to 80 at high effort and 98 at max, which fits the expected pattern if effort levels are working as intended.

Cognition also describes behavioral differences between effort levels that match that step data. Medium moves into action faster, which suits simple and intermediate tasks. High and max plan more, explore more of the codebase, and manage uncertainty through more complex verification, which accounts for the extra steps. The intended use is picking the level that fits the task rather than always running max.

The caveat is that these are Cognition’s measurements, on Cognition’s benchmark, against Cognition’s previous model. The direction is plausible and the mechanism is specific, but nobody outside the company has reproduced the 18-versus-48 figure.

Why a High Score Does Not Settle the Question

A Cursor study published in June 2026 found that 63% of the top-ranked model’s successful resolutions on SWE-bench Pro came from retrieving a known fix from the public web or the evaluation container’s file system, rather than reasoning through the code, according to coverage of the study’s findings. Cursor’s own Composer 2.5 fell 20.7 points, and Cursor said it does not treat its own SWE-bench Pro score as reliable for that reason.

Two mechanisms accounted for most flagged cases. Git-history mining, in 9%, used commands like git log --all against the bundled .git directory to extract the fix that was sitting on disk the whole time.

The structural problem for any mergeability benchmark is that every task is drawn from a real bug later fixed in a public repo, so the answer exists somewhere. FrontierCode’s 1.1 safeguards address this directly. The approach differs from Cursor’s: FrontierCode keeps the internet on and detects misuse rather than blocking access, because several of its tasks require looking up API contracts, and Cognition argues that disabling the internet entirely would understate model performance on tasks where search is a legitimate part of the reasoning workflow.

For your own evaluations, Cursor’s recommendations are worth following: isolate git history before the agent starts, proxy egress through an allowlist restricted to package registries, and audit transcripts with a reviewer who does not see the pass/fail outcome.

Access Lock-In and Where SWE-2 Falls Short

SWE-2 is distributed through Devin, across Desktop, CLI, Web, and Fusion. Cognition shipped it in Devin Desktop and CLI on release day and announced rollout to Devin Web and Fusion alongside it. The announcement included no separate published per-token price and no open-weights release. Teams that want the model run it inside Cognition’s environment.

That is a real cost, though less restrictive than it first appears. SWE-2 is not a raw model you would route your own prompts through; it is post-trained for the Devin agent harness, and the step-count and cost figures were measured inside that harness. Splitting it into a generic API would likely lose much of the reported efficiency. The trade-off is that your migration path runs through Cognition, and total cost of ownership calculations have to include what it costs to leave, not just what you pay per month. We covered that arithmetic in our analysis of total cost of ownership in projects.

Terminal-Bench 4 is the clearest capability gap. It still beats its own Kimi K3 base at 21.5% and Grok 4.6 at 20.3%, but if your work involves hard long-horizon terminal tasks, SWE-2 is not the leading choice. Cognition published that number in its own table, which is worth noting: the vendor showed a benchmark where its model performs worst.

Cognition also published trustworthiness evaluations covering propaganda and censorship behavior, run with a GPT-5.6 Luna judge on a set of questions about politically sensitive topics in China collected by Pan and Xu, submitted in English, Simplified Chinese, and Traditional Chinese. That is a narrow slice of alignment, and Cognition picked both the questions and the judge. It is not a general safety evaluation.

Measuring Cost per Finished Task

Published scores and per-token prices both miss what you actually pay. A model that costs half as much per token but takes three times as many turns to solve a task ends up more expensive. Track tokens, turns, and resolved status per task, then divide total cost by tasks resolved. Cognition’s central claim is a win on exactly this metric, so it is the right thing to measure.

“`python
# Cost per resolved task for a coding agent, from run telemetry.
# Note: prod use should add retry logic, per-run attribution,
# and persistent storage. Rates are per million tokens.
from dataclasses import dataclass

@dataclass
class TaskRun:
task_id: str
model: str
input_tokens: int
output_tokens: int
cache_read_tokens: int
turns: int
wall_seconds: float
resolved: bool # human review or test-suite verdict, not the agent’s claim

def cost_per_resolved_task(runs, in_rate, out_rate, cache_rate=0.0):
total = 0.0
resolved = 0
for r in runs:
total += r.input_tokens / 1e6 * in_rate
total += r.output_tokens / 1e6 * out_rate
total += r.cache_read_tokens / 1e6 * cache_rate
resolved += 1 if r.resolved else 0
if resolved == 0:
return float(“inf”)
return total / resolved

# Run the same task set through SWE-2 medium, high, and max with git history
# stripped and egress proxied, then compare cost per resolved task. If the
# harness lets the agent reach the upstream PR, you are measuring retrieval.
“`

Two details make the comparison honest. First, count a task as resolved only on a human or test-suite verdict, never on the agent’s own claim that it finished. Second, run the same task set through every model you are comparing, because a model tuned for one harness will look worse on another. The step-count numbers Cognition reports were measured inside Devin, so your numbers on your own scaffold will differ.

SWE-2, Fable 5.1, and GPT-6 Astra Compared

All four rows below come from Cognition’s own benchmark table, which is the only source for these numbers. Read them as one vendor’s comparison of its model against three competitors it did not run. The Terminal-Bench 4 column is included because it is where Cognition’s published table shows SWE-2 lagging badly.

The shape of that table tells the main story. SWE-2 is within a point of Fable 5.1 on FrontierCode and beats its own predecessor by eight points, but on Terminal-Bench 4 it trails Fable 5.1 by 28.5 points and GPT-6 Astra by 30.6 points. The cost advantage is clear on the benchmark where Cognition’s model is competitive and much less relevant on the one where it is not.

Verdict

SWE-2 is a credible incremental improvement for teams already running Devin. If Cognition’s numbers hold, medium effort gives better-than-SWE-1.7 results at roughly a fifth of the cost with a third of the turns, which changes what a long agent run costs to finish. The single-run, multi-effort training setup is a solid engineering contribution, and the Pareto-slope argument for choosing the penalty is clearly specified rather than vague.

Three factors limit a clear recommendation. Every headline figure is vendor-reported on a vendor-built benchmark with no independent re-run published. Distribution is Devin-only, with no open weights and no published per-token price. And Terminal-Bench 4, where Cognition’s own table shows poor performance, is the benchmark closest to hard real-world engineering work.

If you want to test the efficiency claim, run medium effort against your own bug backlog with git history isolated and egress restricted, log tokens and turns per ticket, and compute cost per resolved task. Benchmark scores are a starting filter, not a decision.

Sources and References

Sources cited while researching and writing this article:

• SWE-1.7: Frontier Intelligence at a Fraction of the Cost
• FrontierCode Leaderboard | Cognition
• FrontierCode 1.1 | Cognition
• AI Coding Benchmark Scores Are Inflated by Answer Retrieval, Cursor …
• Introducing SWE-2: Pushing the Pareto Frontier | Cognition