Zavo has published Hospital-bench, a benchmark of the administrative work behind a hospital visit, and no AI model fully passed more than 17 of its 48 tasks. The write-up on Zavo’s site is dated 12 November, and Zavo’s Can Zehebi announced the results on X on 20 November. Of the 20 models tested, the best mean score was 62.4%, and 14 of the 48 tasks were passed by none.
Introducing Hospital-bench: the most comprehensive hospital operations simulation built to evaluate AI agents.
— Can Zehebi (@canz101) November 20, 2026
Behind every hospital visit are hours of administrative work: insurance verification, prior authorizations, claims, billing, patient records, and paperwork spread across dozens of workflows.
We built Hospital-bench to test whether AI agents can handle this complexity in a realistic hospital environment.
Here are the results ↓

What Zavo built
The test runs inside a made-up health system. Zavo Health has a 320-bed hospital, four clinics and ten payers, in an invented US state called Merrovia, and the agent plays its operations coordinator. It works through 131 tools in 13 connected apps, from registration and payer portals to claims, referrals, pharmacy and records release. Each app is the record for some facts and keeps stale copies of others, so the systems disagree. The rules sit on a drive of 149 documents with no search, old editions and drafts included.
A task is a few hours of work items. The prompt names no cases, so the agent finds them in the queue, works against a simulated clock that runs from 08:00 to 16:00, and finishes with a handoff for the next coordinator. In 27 places Zavo’s rules deliberately differ from common US practice, so a model that leans on memory acts on the wrong answer. One payer’s approvals last 30 days, for example, not the usual 60 to 90.

How the runs are graded
No language model judges anything. Zavo runs 1,678 checks on the final state of the apps and on what the agent did, and 422 of them cover what the agent must not do. A shared suite of 162 standing safety rules applies on top. A task scores the correct cases and the handoff, minus wrong actions, divided by the number of cases plus one. Any harm sets that task’s score to zero, and a pass means every check holds. The tasks and answers are private, which Zavo says keeps them out of training data.
The results
Across the 20 models, mean scores run from 62.4% down to 4.0%. The leaders sit close together and their 95% intervals overlap. No model passed more than 17 tasks outright, no task was passed by more than 13 models, and 14 tasks were passed by none. Different models pass different tasks, which Zavo says is part of why neighbouring scores overlap.
Cost per task ranges from under a dollar to about nine dollars, and the models that score more don’t always cost more. Zavo prices runs on the Anthropic and OpenAI APIs at list price and uses the billed amount for runs through OpenRouter. Longer tasks are harder for everyone. Scores drop as a task grows from one case to ten, and most models slip again on the three tasks that run into a second session. There the agent has to pick up its own handoff after its earlier tool results have been cleared.
Where the agents slip
Zavo sorts the failures into a handful of patterns:
- Closing work that a payer, medical records or billing still owns.
- Skipping a step the agent has just read. On one task, every model Zavo triaged read the required Medicaid check and skipped it.
- Missing a regulatory clock, such as sending a patient notice at the legal limit when policy asks for it sooner.
- Disclosing records that policy says to withhold from attorneys or agencies.
- Trusting a lagging portal count or an old coverage banner over the system of record.
- Rebilling or voiding claims that were already paid, which Zavo grades as harm.

What Zavo takes from it
Zavo says the best models do well once they’ve found the facts, and stumble on corrections recorded somewhere else, look-alike records, deadlines, safety rules and long multi-case work. It points to jobs agents can take on now: checking eligibility, assembling prior authorisations, drafting appeals and patient estimates, and triaging the queue, with a coordinator reviewing every action that changes a record or reaches a patient. Teams can ask for the environment, sample tasks and graded runs through [email protected].
Sources
- Zavo: Hospital-bench, measuring how well AI agents run hospital operations
- Can Zehebi on X: Hospital-bench announcement and results thread
Discover more from Sulat AI & Tech News
Subscribe to get the latest posts sent to your email.




