Sulat.com / Writing on AIFriday, November 20, 2026
Artificial Intelligence/19 November 2026

Ramp’s SWE-Bench v2 grades coding agents with LLM judges on 500 production tasks

Ramp Labs has published SWE-Bench v2: 500 tasks from code its engineers shipped, graded by LLM judges instead of tests, with each lab's models run in their own harness. The top four sit within a few points.

Glossy 3D Ramp logo beside a balance scale weighing code patches and a podium of leaderboard bars

At a glance

  • Ramp SWE-Bench v2 has 500 tasks built from code that more than 200 Ramp engineers shipped, six times as many as v1.
  • LLM judges now grade solutions against the engineer-reviewed patch, and two judges' agreement rose from 54.2% to 93.6% after deliberation.
  • On Ramp's leaderboard Carbon solves 75.4% of tasks at US$4.25 each and Claude Fable 5.2 solves 73.4% at US$3.79.
  • Ramp says native harnesses generally raise both solve rates and costs compared with a bash-only toolkit.

Ramp Labs has published Ramp SWE-Bench v2, a coding benchmark built from 500 tasks that more than 200 Ramp engineers shipped to production. It grades with LLM judges instead of tests, and it runs each lab’s models in that lab’s own agent harness. Ramp announced it on X on 18 November and put the leaderboard and methodology on its Labs site.

Ramp says the top four models sit within a few points of each other. Carbon, Google’s Gemini 4 model running in OpenCode, solves 75.4% of tasks. Claude Fable 5.2 in Claude Code solves 73.4%.

Where the tasks come from

Every task starts as a change that Inspect, Ramp’s in-house background coding agent, shipped to production after an engineer reviewed it. The prompt is written from what the engineer asked Inspect to do, and the merged pull request becomes the reference answer, which Ramp calls the gold patch. Ramp’s engineering post says v2 has six times as many tasks as v1, which had 80. Each model gets one attempt per task at high effort, and Ramp scores the share it solves on that first try.

Inspect is the agent Ramp offered to onboard teams of 100-plus engineers onto, as covered in an earlier piece.

LLM judges replace the tests

In v1, Ramp graded solutions by running the tests merged with each gold patch. Its engineering post says those tests often rejected valid answers because they expected one particular implementation, and filtering those tasks out left tests that missed requirements. In v2, a judge model reads the task prompt, the repository context, the gold patch and the candidate patch, then decides whether the two do the same thing.

Ramp tested the idea on 72 v1 tasks, with seven judge models grading solutions from twenty solver models. A majority vote across the judges tracked the test-based pass rates with a correlation of 0.85, and all seven judges were stricter than the tests. They disagreed with each other, though. One judge passed nearly 60% of comparisons while others passed about 25%. Ramp’s fix was to split the job in two. The judges first list the differences between the patches, then decide whether each difference matters to the task.

On 360 comparisons, two judges with opposite habits agreed on 54.2% of grades in the original panel. They agreed on 76.4% when voting independently in the new pipeline, and on 93.6% after deliberating. Ramp then asked the engineers behind the original pull requests to check the verdicts, and they agreed with nearly 90% of the task-level calls. Ramp also found no sign that a model scored better for sharing a provider with the model that wrote the gold patch.

Bar chart: two LLM judges agree 54.2% at first, 76.4% on first vote, 93.6% after deliberation
Two of Ramp’s judge models went from agreeing on about half of grades to nearly nine in ten once they deliberated. Source: Ramp.

What the leaderboard shows

Ramp says performance generally scales with model size and test-time compute, but the returns compress quickly. Fable 5.2 solves 73.4% of tasks at a mean cost of US$3.79 each. Opus 5.5 solves 67.8% at US$1.53, Sonnet 5.5 solves 63.2% at US$0.47 and Haiku 5.5 solves 57.4% at US$0.28. DeepSeek’s V4.1 Flash solves 60.8% for US$0.22. At the other end, Muse Wonder costs US$4.59 a task for 58.6%, and Grok 4.7 costs US$4.32 for 64.4%.

Bars of solve rate and cost per task for ten models; Carbon 75.4% at $4.25, Opus 5.5 67.8% at $1.53
Solve rate climbs with spend, but not in a straight line: Opus 5.5 gets 67.8% for US$1.53 a task. Source: Ramp SWE-Bench v2.

How the models spend their effort

Carbon averages 137 turns, 349 tool calls and 72K output tokens per task, and takes 26.3 minutes. Opus 5.5 averages 39 turns, 40 tool calls and 16K output tokens, in 14.7 minutes. Sonnet 5.5 takes 26 turns, the fewest of any model on the board. Ramp says Carbon generates about 4x the output tokens of Opus 5.5 and that Anthropic models take the fewest turns to finish a task.

Bars comparing Carbon, Opus 5.5 and Sonnet 5.5 on turns, tool calls and output tokens per task
Carbon takes 137 turns and 349 tool calls a task; Opus 5.5 takes 39 turns and 40 calls. Source: Ramp SWE-Bench v2.

The harness matters

Ramp runs every model two ways. One run uses the lab’s own harness, such as Claude Code, OpenCode, Muse Code or Grok Build. The other uses mini-swe-agent with only a bash tool, which isolates the model from the harness. Ramp says native harnesses generally raise both solve rates and costs compared with bash alone, but the effect isn’t uniform across models.

The tasks are private

Ramp’s methodology page says it can’t release prompts, patches, tests or repository states, so only aggregate numbers, pairwise comparisons and redacted examples are public. Ramp says it spends millions of dollars on tokens for its own engineers and for customers through Router, and that it built the benchmark to decide where that money goes.

Hero illustration uses the official Ramp mark, which remains the trademark of its owner.


Discover more from Sulat AI & Tech News

Subscribe to get the latest posts sent to your email.