Sulat.com / Writing on AITuesday, November 17, 2026
Artificial Intelligence/17 November 2026

Baseten’s guide says to test open coding models on your own merged pull requests

Baseten's guide to evaluating open models for coding agents says to benchmark them on about 30 of your own merged PRs and track cache-adjusted cost per successful task. Its public page lists no scores or prices.

Glossy 3D Baseten logo beside a laptop of code, a fanned deck of diff cards and a scale weighing coins against a green checkmark.

At a glance

  • Baseten says its guide helps teams find which coding tasks open models can handle, by benchmarking them against about 30 of the team's own merged pull requests instead of relying on generic benchmarks.
  • The process starts with a quality bar set by the closed model a team already trusts, then a shortlist of open models and a pilot that measures quality, cost and speed.
  • Baseten says teams should keep their existing harness, such as Claude Code, Codex, Cursor or OpenCode, so the model is the only thing that changes.
  • Baseten names cache-adjusted cost per successful task as the key metric, saying cost per token hides the retries and extra reasoning that some models need.
  • The full guide sits behind a Download now button, and Baseten's public page lists no benchmark scores, model names or prices.

Not every coding task needs a frontier model, Baseten says, and it has published a guide on how to find out which ones don’t. Its advice: run open models against about 30 of your own merged pull requests, not generic benchmarks. Baseten sells hosting for open models through its Model APIs and Dedicated Inference.

Alex Ker, Chloe Florit and Rachel Rapp wrote the guide. Its page carries a 21 October 2026 date, and Baseten pushed it on X on 16 November with four bullet points: pick a pilot team and workload, measure quality on your own merged PRs, track cache-adjusted cost per successful task, and keep your harness while using a map from today’s models to open alternatives.

What the guide recommends

Coding agents have become one of the fastest-growing costs in engineering, Baseten says, yet most teams still run a frontier model on every task. The guide targets engineering leaders, platform teams and anyone who owns AI coding spend.

Step one is a quality bar, set with the closed model your team already trusts. Step two is a shortlist of open models that could match it. After that you benchmark them on those 30 or so merged PRs and scope a pilot that collects evidence on quality, cost and speed.

Four steps: set a quality bar, shortlist open models, run about 30 merged PRs, compare quality, cost and speed. The harness stays.
Four steps from a quality bar to a pilot, with your harness unchanged.

Keep your tools, change the model

Baseten tells teams not to switch tools. Keep the harness you have, the guide says. That’s the agent software wrapped around the model, whether it’s Claude Code, Codex, Cursor, OpenCode or something built in-house. With only the model changing, results are easier to measure and engineers are easier to win over, Baseten says. It made the same point on 15 October, when it added Meta’s Muse Spark Open: Baseten’s Model APIs use an OpenAI-compatible endpoint, so a coding agent can point at the model without any change to the harness.

Cost per successful task

Baseten names cache-adjusted cost per successful task as the metric that matters most. Cost per token hides the retries and extra reasoning that some models need, it says.

Cost per token counts tokens only; cost per successful task adds retries, extra reasoning and a cache adjustment.
Baseten counts retries and extra reasoning, not just the token price.

Open models don’t replace closed ones

The guide isn’t arguing for dropping closed models. Most teams end up with a mixed fleet, Baseten says: frontier closed models on the hardest, longest-running work, open models on high-volume jobs like migrations, refactors and bug fixes.

Closed frontier models take the hardest, longest-running work; open models take migrations, refactors and bug fixes.
Baseten’s split: frontier models for the hard work, open models for the volume.

The full guide sits behind a Download now button. What’s public is a summary and a short FAQ, with no benchmark scores, model names or prices. Baseten’s guide page also carries a banner for a 28 October webinar on the framework, hosted by Alex Ker with Molly Elfers, a Baseten solutions architect.

Sources


Discover more from Sulat AI & Tech News

Subscribe to get the latest posts sent to your email.