Cognition Launches FrontierCode 2.0 Coding Benchmark Testing Pull Request Mergeability Across Underspecified Tasks
Introducing FrontierCode 2.0: our new coding benchmark that measures a model’s ability to go beyond the spec. Most coding evals measure a model’s ability to write code that meets a spec. With FrontierCode 1.0, we asked a new question: would you actually merge the PR? FrontierCode 2.0 is comprised of more diverse tasks, featuring larger scopes and higher mergeability standards. We are now asking: not only would you merge this PR, but should you? One key feature of FrontierCode 2.0 is underspecified prompts. Rather than including every engineering decision, prompts are deliberately ambiguous, giving the models room to demonstrate their judgment. The result is a benchmark that is far from saturated, with the top model scoring just 45%.
Context
By using deliberately underspecified prompts crafted by open source maintainers and zeroing out solutions that violate unstated codebase invariants, FrontierCode 2.0 exposes why models that ace traditional unit test benchmarks still require human oversight in production software engineering.
Cognition introduced FrontierCode 1.0 in June 2026 and FrontierCode 1.1 in July 2026 to evaluate whether autonomous coding agents produce pull requests that human open source maintainers would actually merge into production repositories. As frontier models began clustering near 50 percent on earlier suites, Cognition rebuilt the benchmark with broader repository coverage and stricter blocking rubrics.
Deep dive
On October 27, Devin creator Cognition unveiled FrontierCode 2.0, a major overhaul of its production coding benchmark designed to test whether AI agents exercise sound software engineering judgment on underspecified tasks. Built by more than 20 open source maintainers who each spent over 40 hours per task, the private 132 task suite draws from 55 percent more repositories than FrontierCode 1.1, adds 43 percent more blocking criteria, and requires reference changes that touch 67 percent more files and 89 percent more lines of code across 25 percent larger codebases. During quality control audits, Cognition's review agents improved on the original upstream pull request in 80 percent of tasks and uncovered a live upstream security vulnerability.
Rather than spelling out every implementation detail, FrontierCode 2.0 uses intentionally ambiguous prompts to test whether models respect implicit architectural invariants. In one example from the Mock Service Worker JavaScript library, models are asked to fix a bug where resetHandlers fails to restart a generator. While Claude Opus 5.5 fixes the immediate symptom by discarding the old generator reference, doing so prevents user cleanup logic inside a finally block from executing, failing the blocking rubric.
On the inaugural leaderboard, Anthropic's Claude Opus 5.5 takes first place at 45.5 percent at an average rollout cost of 12.80 dollars, followed by OpenAI's GPT 6 Astra at 42.6 percent for 6.67 dollars and Claude Sonnet 5.5 at 40.8 percent. OpenAI's GPT 6.1 Sol leads the cost efficiency tier at 38.9 percent for just 1.03 dollars per task, ahead of Claude Fable 5.1 at 32.4 percent, xAI's Grok 4.7 at 28.5 percent, Gemini 4 Argon at 26.3 percent, Claude Haiku 5.5 at 24.6 percent, Meta's Muse Wonder 1.0 at 24.4 percent, and Cognition's SWE 2 at 23.9 percent. Chinese open weight models Kimi K3 and GLM 5.3 score 17 percent, while US open weight models Inkling Small and Beam 1.0 trail at roughly 3 percent.
What to expect next
Because FrontierCode 2.0 sharply separates frontier reasoning models from mid tier and open weight checkpoints that previously clustered together on DeepSWE v1.1 and FrontierCode 1.1, enterprise engineering teams now have a clear reference for weighing rollout cost against mergeable pull request quality. Watch whether upcoming agent harnesses integrate automated cleanup and regression verifiers to overcome the blocking rubric failures that currently cap even Claude Opus 5.5 below 46 percent.