Sulat.com / Writing on AITuesday, November 17, 2026
Agents/17 November 2026

Databricks says Genie passed 91% of benchmark runs; two unnamed frontier agents got 45% and 58%

Jeff Fletcher and Rishabh Singh say Genie also used less than half the tokens. Databricks ran the benchmark itself and doesn't name Agent A or Agent B.

Glossy red 3D stacked Databricks logo surrounded by glass panels showing a notebook, a bar-chart dashboard and a saved query, with a document lifted out.

At a glance

  • Databricks says Genie passed 91% of runs on a benchmark built with Turing, against 45% for Agent A and 58% for Agent B, two frontier agents the company doesn't name.
  • According to Databricks, Genie averaged 431,829 tokens a run against 889,019 and 1,051,070 for the others, and finished about four times faster than Agent A and three times faster than Agent B.
  • The benchmark uses a fictional UK energy supplier, Calderwick Energy, with hundreds of tables plus notebooks, dashboards and saved queries; each agent ran every task five times in one live Databricks workspace.
  • In Databricks' worked example, a table showed 0.31 p/kWh for FY24/25 while a notebook recorded 0.52; Genie answered 0.52 in all five runs and the other two agents didn't get the correct rate across their runs.
  • Databricks ran the benchmark itself, doesn't give the task count, and says the gap may be smaller in new deployments or when tables alone can answer the question.

Genie, Databricks’ data agent, passed 91% of runs on a new enterprise benchmark. Two frontier agents that Databricks doesn’t name passed 45% and 58%. The company’s blog post, dated 11 November, adds that Genie used less than half the tokens and finished three to four times faster. Databricks built the test with Turing and sells Genie, so every figure here is its own.

The post is by Jeff Fletcher and Rishabh Singh. On 16 November, US time, Databricks’ X account pushed it with one line of explanation: a valid SQL query can still return the wrong answer when it uses the wrong definitions or calculation rules.

What the benchmark covers

Most data agent benchmarks stick to tables and the pipeline code behind them, and Databricks says agents score about the same on them. This one also feeds in notebooks, dashboards, saved SQL queries and documents, where companies keep the definitions and calculation rules that a schema leaves out. Genie Ontology is Databricks’ name for the layer that extracts that knowledge and passes the relevant pieces to Genie; the company says it’s now generally available.

The test company is invented. Calderwick Energy is a UK electricity and gas supplier with hundreds of tables, and the biggest holds about 120 million rows of half-hourly meter data. Databricks seeded the workspace with the clutter of a real one: copied notebooks, abandoned analyses, stale documents, dev copies and reporting tables nobody had touched in months. Every agent got each task five times in the same live workspace. A run passed only if it cleared a grading rubric, and any tolerance for small numeric differences was fixed beforehand.

Bar chart of pass rates per run: Genie 91%, Agent B 58%, Agent A 45%, each agent run five times per task.
Genie passed 91% of runs; the two unnamed agents passed 58% and 45%.

How the agents compare

Genie ran natively. Databricks wired the other two through a toolbelt of its own and left out the Genie MCP, since that would have handed them the Genie features being tested. It calls them Agent A and Agent B, describes them as agents from frontier AI labs, and doesn’t name them or their models.

AgentPass rateAvg wall time per runAvg tokens per run
Genie91.0%100.3431,829
Agent A45.0%434.6889,019
Agent B58.0%306.51,051,070
Figures as published in the Databricks blog post. The post gives no unit for wall time.

The post’s title promises 1.6 times the accuracy. That figure compares Genie with Agent B, 91% to 58%; against Agent A it’s closer to double, 91% to 45%. The +46% and +33% in the text are percentage-point gaps.

Three cards of average tokens per run: Genie 431,829, Agent A 889,019 about 4x slower, Agent B 1,051,070 about 3x slower.
Databricks says the other two agents used more than twice Genie’s tokens.

The example Databricks walks through

A task asks for the planning rate Calderwick’s Commercial team set for FY24/25. In the bill cost stack table, the FY24/25 rows show 0.31 pence per kWh under a column called cfd_rate_p. The notebook that builds that table explains that the mart applies FY25/26 assumptions even to earlier billing periods, and its planning rates by year section puts FY24/25 at 0.52 p/kWh.

According to Databricks, Genie Ontology found a snippet with the historical rates and linked it to the source notebook, 45_mart_bill_cost_stack. Genie confirmed it there and answered 0.52 first time, in all five runs. One of the other agents said 0.31 every time. The second gave 0.31, doubted that a year-specific rate was documented, or did both.

The table shows 0.31 p/kWh for FY24/25 while the notebook records 0.52; Genie answered 0.52 in all five runs.
The table held the wrong year’s rate; the notebook held the right one.

What Databricks says limits the result

The benchmark is aimed at agents working in an established Databricks deployment, where Unity Catalog governs the data and workspace assets hold the business knowledge. Databricks writes that in new deployments, or where a table and its schema can answer the question, “the difference between agents may be smaller.”

The post doesn’t give the number of tasks. It says the tasks and assets were designed for sharing and points readers to their Databricks account team to run the benchmark on their own workspace. Databricks files it under its Week of Genie, with more news promised on Genie, Genie Agents and Genie MCP.

Sources


Discover more from Sulat AI & Tech News

Subscribe to get the latest posts sent to your email.