Sulat.com / Writing on AIThursday, November 19, 2026
Agent Engineering/18 November 2026

Runta and Baseten find one model costs up to 5x more per task depending on the coding harness

Runta's FrontierHarness Eval v1.1 ran 22 coding agent harnesses with five open models across 27,280 runs. The tests use open models served by Baseten only, and each pass rate carries a margin of about 10 points.

3D orange fox logo beside five glass boxes with blank price tags hanging at different heights in a server room

At a glance

  • Runta says its FrontierHarness Eval v1.1, run with Baseten, found the most expensive harness cost 1.8 to 5.0 times as much per task as the cheapest with the model held fixed.
  • The widest gap was with DeepSeek V4.1 Flash, where Cline CLI cost 5.0 times as much as OpenCode v1 and passed 2.0 points more tasks, Runta says.
  • The best-value harness differed for each of the five models, and the median harness moved 11 of 22 places in cost rank between models, according to Runta.
  • Runta says 64.5% of the list-price cost went to runs that didn't pass, and that prompt size didn't predict cost per pass while the number of steps did.
  • The tests use open models served by Baseten only, and each pass rate carries a 95% interval of about 10 points over 62 tasks, Runta says.

Runta and Baseten say the same model can cost up to five times more per task depending on the coding agent harness it runs in. A harness is the program that gives a model its tools, instructions and loop, and the widest gap in their FrontierHarness Eval v1.1 came with DeepSeek V4.1 Flash, where Cline CLI cost 5.0 times as much as OpenCode v1 and passed 2.0 points more tasks.

Runta, which sells an execution layer for agents, published the benchmark on 17 November. The New Stack covered it under the headline “Your AI coding agent could cost 5x more depending on the harness, benchmark finds”. Baseten announced the results on X and put the spread at up to 4.4 times on cost per pass:

What Runta tested

The run covers 22 harness configurations, five open models, 62 terminal tasks and four attempts per task, which comes to 27,280 runs and 83 billion tokens. The 22 come from 18 harnesses: the four DeepSeek Harness profiles and two versions of OpenCode each count separately. Baseten, which announced a voice agent stack the day before, served GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash, Muse Spark Open and Kimi-K3.

Each harness ran at its default settings and maximum reasoning effort through the Harbor evaluation framework, with task time limits from 15 minutes to 8 hours. Every trial got a fresh Runta runtime that was deleted afterwards.

Runta reports two cost measures: average cost per task, and cost per pass, which is that average divided by the pass rate. Both are Baseten’s list prices multiplied by the tokens each harness reported, so they’re estimates, not invoices. Runta flagged 216 trials for reward hacking, 163 of them online lookups, and scored them as failures.

How big the gap is

The model sets most of the bill. Runta’s analysis puts 92.8% of the variation in cost per task on the model, 2.3% on the harness alone and 4.9% on the particular pairing. With the model fixed, the most expensive harness still cost between 1.8 and 5.0 times as much per task as the cheapest. Runta shuffled trials between harnesses 300 times to test for chance, which produced ratios of 1.2 to 1.8 times. All five real ratios were larger (p = 0.003).

Bar chart: with DeepSeek V4.1 Flash, Cline CLI costs 5.0x OpenCode v1 per task for 2.0 points more passes
With DeepSeek V4.1 Flash fixed, Cline CLI cost 5.0 times OpenCode v1 per task for 2.0 more points of pass rate.

The two companies quote different per-pass maximums: Runta’s post puts the spread at 1.9 to 4.9 times, and Baseten’s X post says up to 4.4 times. With the other four models, the cheapest harness per task was at most 0.4 points behind the most expensive one.

The cheapest harness changes with the model

Cost ranks moved a lot between models. The median harness shifted 11 of 22 places by cost per task. Codex CLI was the cheapest with GLM-5.3 and 16th with Kimi-K3. DeepSeek Harness: Creator went from 1st with Muse to 21st with GLM Flash, and OpenCode v1 was 1st with DeepSeek and 13th with Muse.

Cost rank flips: Codex CLI 1st with GLM-5.3 and 16th with Kimi-K3; OpenCode v1 1st to 13th; DeepSeek Harness Creator 1st to 21st
A harness that’s cheapest with one model can rank near the bottom with another.

Across all five models, four harnesses sit on the line that no other harness beats on both cost and pass rate: OpenCode v2 (lowest average cost per task, US$0.561), Pi, Codex CLI and ZCode (highest pass rate, 61.9%). Codex CLI had the lowest cost per pass overall at US$0.937 and Antigravity CLI the highest at US$1.63. Runta’s leaderboard ranks Claude Code 2.1.308 21st of 22 by cost per pass, at US$1.55, with these open models.

Runta picks a best-value harness for each model: the lowest cost per pass, unless another harness costs at most 30% more per pass and passes at least 5 points more often.

ModelBest-value harnessShare of 4,000 resamples it won
GLM-5.3Codex CLI99.6%
GLM-5.3 FlashPi42%
DeepSeek V4.1 FlashZCode47%
Muse Spark OpenOh My Pi53%
Kimi-K3Alma25%
Runta redrew the 62 tasks 4,000 times and re-picked the best-value harness each time.

What drives the cost

Simpler harnesses don’t cost less. Prompt size, tool count and cache hit rate didn’t track cost per pass, and the number of steps did, with a rank correlation of +0.51 across the 22 harnesses (p = 0.015). Each step sends the context again, so a trial used 2.0 to 4.8 million input tokens on average. mini-SWE-agent has the smallest first request at 882 tokens, took 38 steps per pass and ranked 17th of 22 on cost per pass.

Most of the money went to runs that didn’t pass: 64.5% of the list-price cost, with 52.7% on failures and 11.8% on timeouts. About three in four failed trials from the five best-value harnesses ended on a final answer instead of the time limit, from 68% with Pi to 82% with Codex CLI.

Proportion bar of benchmark cost: 35.5% passed, 52.7% failed, 11.8% timed out, so 64.5% went to runs that did not pass
Runta puts 64.5% of the benchmark’s list-price cost on runs that didn’t pass.

What the benchmark covers

The tests use open models only, all served by Baseten. Runta says closed models can give different results and that another provider’s cache and prices can change the cost ranks. It plans to add Claude Opus 5.5, GPT-6 Astra and Gemini 4 Carbon. Runta and Baseten sponsor the leaderboard, and the trials ran on Runta’s runtime.

Each pass rate carries a 95% interval of about 10 points over 62 tasks, so Runta treats a gap of a few points as a tie. Four of its five best-value picks are the likeliest of several close harnesses, and Alma with Kimi-K3 won only 25% of the resamples. Runta didn’t test whether a check before a harness stops would cut the spend on failed runs.

Runta’s advice is to test candidate harnesses with your own model on a private set of your own tasks. The FrontierHarness leaderboard is public, the trial jobs are stored on Harbor Hub, and Runta lists closed models and long-horizon work as its next steps.

Sources


Discover more from Sulat AI & Tech News

Subscribe to get the latest posts sent to your email.