Ramp Labs has published Ramp SWE-Bench v2, a coding benchmark built from 500 tasks that more than 200 Ramp engineers shipped to production. It grades with LLM judges instead of tests, and it runs each lab’s models in that lab’s own agent harness. Ramp announced it on X on 18 November and put the leaderboard and methodology on its Labs site.
Ramp says the top four models sit within a few points of each other. Carbon, Google’s Gemini 4 model running in OpenCode, solves 75.4% of tasks. Claude Fable 5.2 in Claude Code solves 73.4%.
Where the tasks come from
Every task starts as a change that Inspect, Ramp’s in-house background coding agent, shipped to production after an engineer reviewed it. The prompt is written from what the engineer asked Inspect to do, and the merged pull request becomes the reference answer, which Ramp calls the gold patch. Ramp’s engineering post says v2 has six times as many tasks as v1, which had 80. Each model gets one attempt per task at high effort, and Ramp scores the share it solves on that first try.
Inspect is the agent Ramp offered to onboard teams of 100-plus engineers onto, as covered in an earlier piece.
LLM judges replace the tests
In v1, Ramp graded solutions by running the tests merged with each gold patch. Its engineering post says those tests often rejected valid answers because they expected one particular implementation, and filtering those tasks out left tests that missed requirements. In v2, a judge model reads the task prompt, the repository context, the gold patch and the candidate patch, then decides whether the two do the same thing.
Ramp tested the idea on 72 v1 tasks, with seven judge models grading solutions from twenty solver models. A majority vote across the judges tracked the test-based pass rates with a correlation of 0.85, and all seven judges were stricter than the tests. They disagreed with each other, though. One judge passed nearly 60% of comparisons while others passed about 25%. Ramp’s fix was to split the job in two. The judges first list the differences between the patches, then decide whether each difference matters to the task.
On 360 comparisons, two judges with opposite habits agreed on 54.2% of grades in the original panel. They agreed on 76.4% when voting independently in the new pipeline, and on 93.6% after deliberating. Ramp then asked the engineers behind the original pull requests to check the verdicts, and they agreed with nearly 90% of the task-level calls. Ramp also found no sign that a model scored better for sharing a provider with the model that wrote the gold patch.

What the leaderboard shows
Ramp says performance generally scales with model size and test-time compute, but the returns compress quickly. Fable 5.2 solves 73.4% of tasks at a mean cost of US$3.79 each. Opus 5.5 solves 67.8% at US$1.53, Sonnet 5.5 solves 63.2% at US$0.47 and Haiku 5.5 solves 57.4% at US$0.28. DeepSeek’s V4.1 Flash solves 60.8% for US$0.22. At the other end, Muse Wonder costs US$4.59 a task for 58.6%, and Grok 4.7 costs US$4.32 for 64.4%.

How the models spend their effort
Carbon averages 137 turns, 349 tool calls and 72K output tokens per task, and takes 26.3 minutes. Opus 5.5 averages 39 turns, 40 tool calls and 16K output tokens, in 14.7 minutes. Sonnet 5.5 takes 26 turns, the fewest of any model on the board. Ramp says Carbon generates about 4x the output tokens of Opus 5.5 and that Anthropic models take the fewest turns to finish a task.

The harness matters
Ramp runs every model two ways. One run uses the lab’s own harness, such as Claude Code, OpenCode, Muse Code or Grok Build. The other uses mini-swe-agent with only a bash tool, which isolates the model from the harness. Ramp says native harnesses generally raise both solve rates and costs compared with bash alone, but the effect isn’t uniform across models.
The tasks are private
Ramp’s methodology page says it can’t release prompts, patches, tests or repository states, so only aggregate numbers, pairwise comparisons and redacted examples are public. Ramp says it spends millions of dollars on tokens for its own engineers and for customers through Router, and that it built the benchmark to decide where that money goes.
Hero illustration uses the official Ramp mark, which remains the trademark of its owner.
Discover more from Sulat AI & Tech News
Subscribe to get the latest posts sent to your email.




