← Wednesday, November 18 deck

39Research1 source posting

Ramp Labs Publishes Ramp SWE Bench v2, 500 Coding Tasks Built From Its Own Production Work

We’re publishing Ramp SWE-Bench v2, featuring 500 tasks built from work shipped to production by over 200 of our engineers, more rigorous verification, and frontier models running in their native harnesses.

Ramp SWE Bench v2 bar chart of pass@1 scores on 500 tasks, led by GPT 6.2 Astra at 76.4 percent, Carbon at 75.4 and Fable 5.2 at 73.4.

Context

Why it matters

A coding benchmark built from code that actually shipped at a finance software company, run in each lab's own agent harness, shows how agents handle real business software better than puzzle style tests do.

Ramp Labs, part of the finance software company Ramp, published the second version of its SWE Bench style coding benchmark. It says the 500 tasks are built from work shipped to production by more than 200 Ramp engineers, with more rigorous verification. Each frontier model runs in its native harness, such as Claude Code, Codex or OpenCode.

Deep dive

Ramp Labs posted that it is publishing Ramp SWE Bench v2, featuring 500 tasks built from work shipped to production by over 200 of its engineers, more rigorous verification, and frontier models running in their native harnesses. The chart attached to the post reports pass@1 at high effort with 95% confidence intervals.

According to the chart and the results page, OpenAI's GPT 6.2 Astra in Codex leads at 76.4%, followed by Google's Carbon in OpenCode at 75.4%, with Claude Fable 5.2 in Claude Code and GPT 6.2 Sol in Codex tied at 73.4%. Hydrogen follows at 71.6% and GPT 6.1 Terra at 70.4%. Further down are Claude Opus 5.5 at 67.8%, Grok 4.7 in Grok Build at 64.4%, Claude Sonnet 5.5 at 63.2% and DeepSeek V4.1 Flash at 60.8%, with Muse Wonder, Haiku 5.5, GPT 6 Luna and Muse Ember between 51.8% and 58.6%.

The results page also breaks scores down by task difficulty and lists cost and time per task. On the hardest quarter of tasks, Carbon solves 43% and Hydrogen 41%, ahead of Grok 4.7 at 36% and GPT 6.2 Astra at 30%. Mean cost per task ranges from $0.07 for GPT 6 Luna to $4.59 for Muse Wonder, and GPT 6.2 Astra finishes a task in 6.5 minutes on average against 26.3 minutes for Carbon.

These are Ramp's own measurements on its own codebase. The post came hours before Google's Gemini 4 Neon launch, so Neon is not included.

What to expect next

Watch whether Ramp adds Gemini 4 Neon and Grok 4.8 once they are widely available, and whether it releases the tasks or harness setup so others can reproduce the scores. Because the tasks come from one company's financial software, other teams may test whether the ranking holds on their own code. The gap between models on the hardest tasks, and the wide spread in cost and time, will matter to teams choosing a coding agent for production work.