Runta and Baseten say the same model can cost up to five times more per task depending on the coding agent harness it runs in. A harness is the program that gives a model its tools, instructions and loop, and the widest gap in their FrontierHarness Eval v1.1 came with DeepSeek V4.1 Flash, where Cline CLI cost 5.0 times as much as OpenCode v1 and passed 2.0 points more tasks.
Runta, which sells an execution layer for agents, published the benchmark on 17 November. The New Stack covered it under the headline “Your AI coding agent could cost 5x more depending on the harness, benchmark finds”. Baseten announced the results on X and put the spread at up to 4.4 times on cost per pass:
Is there a best harness for each popular open model?
— Baseten (@baseten) November 17, 2026
We partnered with @runta to build FrontierHarness Eval v1.1, built by running 83B tokens across 22 harnesses and 5 open models. We found that cost per pass can vary by up to 4.4x across harnesses.
If you’re running a coding agent with open models, maintain your own evals and re-test your harness on new model releases.
What Runta tested
The run covers 22 harness configurations, five open models, 62 terminal tasks and four attempts per task, which comes to 27,280 runs and 83 billion tokens. The 22 come from 18 harnesses: the four DeepSeek Harness profiles and two versions of OpenCode each count separately. Baseten, which announced a voice agent stack the day before, served GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash, Muse Spark Open and Kimi-K3.
Each harness ran at its default settings and maximum reasoning effort through the Harbor evaluation framework, with task time limits from 15 minutes to 8 hours. Every trial got a fresh Runta runtime that was deleted afterwards.
Runta reports two cost measures: average cost per task, and cost per pass, which is that average divided by the pass rate. Both are Baseten’s list prices multiplied by the tokens each harness reported, so they’re estimates, not invoices. Runta flagged 216 trials for reward hacking, 163 of them online lookups, and scored them as failures.
How big the gap is
The model sets most of the bill. Runta’s analysis puts 92.8% of the variation in cost per task on the model, 2.3% on the harness alone and 4.9% on the particular pairing. With the model fixed, the most expensive harness still cost between 1.8 and 5.0 times as much per task as the cheapest. Runta shuffled trials between harnesses 300 times to test for chance, which produced ratios of 1.2 to 1.8 times. All five real ratios were larger (p = 0.003).

The two companies quote different per-pass maximums: Runta’s post puts the spread at 1.9 to 4.9 times, and Baseten’s X post says up to 4.4 times. With the other four models, the cheapest harness per task was at most 0.4 points behind the most expensive one.
The cheapest harness changes with the model
Cost ranks moved a lot between models. The median harness shifted 11 of 22 places by cost per task. Codex CLI was the cheapest with GLM-5.3 and 16th with Kimi-K3. DeepSeek Harness: Creator went from 1st with Muse to 21st with GLM Flash, and OpenCode v1 was 1st with DeepSeek and 13th with Muse.

Across all five models, four harnesses sit on the line that no other harness beats on both cost and pass rate: OpenCode v2 (lowest average cost per task, US$0.561), Pi, Codex CLI and ZCode (highest pass rate, 61.9%). Codex CLI had the lowest cost per pass overall at US$0.937 and Antigravity CLI the highest at US$1.63. Runta’s leaderboard ranks Claude Code 2.1.308 21st of 22 by cost per pass, at US$1.55, with these open models.
Runta picks a best-value harness for each model: the lowest cost per pass, unless another harness costs at most 30% more per pass and passes at least 5 points more often.
| Model | Best-value harness | Share of 4,000 resamples it won |
|---|---|---|
| GLM-5.3 | Codex CLI | 99.6% |
| GLM-5.3 Flash | Pi | 42% |
| DeepSeek V4.1 Flash | ZCode | 47% |
| Muse Spark Open | Oh My Pi | 53% |
| Kimi-K3 | Alma | 25% |
What drives the cost
Simpler harnesses don’t cost less. Prompt size, tool count and cache hit rate didn’t track cost per pass, and the number of steps did, with a rank correlation of +0.51 across the 22 harnesses (p = 0.015). Each step sends the context again, so a trial used 2.0 to 4.8 million input tokens on average. mini-SWE-agent has the smallest first request at 882 tokens, took 38 steps per pass and ranked 17th of 22 on cost per pass.
Most of the money went to runs that didn’t pass: 64.5% of the list-price cost, with 52.7% on failures and 11.8% on timeouts. About three in four failed trials from the five best-value harnesses ended on a final answer instead of the time limit, from 68% with Pi to 82% with Codex CLI.

What the benchmark covers
The tests use open models only, all served by Baseten. Runta says closed models can give different results and that another provider’s cache and prices can change the cost ranks. It plans to add Claude Opus 5.5, GPT-6 Astra and Gemini 4 Carbon. Runta and Baseten sponsor the leaderboard, and the trials ran on Runta’s runtime.
Each pass rate carries a 95% interval of about 10 points over 62 tasks, so Runta treats a gap of a few points as a tie. Four of its five best-value picks are the likeliest of several close harnesses, and Alma with Kimi-K3 won only 25% of the resamples. Runta didn’t test whether a check before a harness stops would cut the spend on failed runs.
Runta’s advice is to test candidate harnesses with your own model on a private set of your own tasks. The FrontierHarness leaderboard is public, the trial jobs are stored on Harbor Hub, and Runta lists closed models and long-horizon work as its next steps.
Sources
- Introducing FrontierHarness Eval v1.1, Runta, 17 November 2026
- FrontierHarness Eval leaderboard, v1.1
- Baseten on X, 17 November 2026
- Your AI coding agent could cost 5x more depending on the harness, benchmark finds, The New Stack
Discover more from Sulat AI & Tech News
Subscribe to get the latest posts sent to your email.




