The cheapest model that can answer a question is not always the cheapest system that can answer it well. A new Hightouch experiment shows the difference: let a fast decision model handle the routine cases, then send uncertain work to a frontier model.
The task was grading identity resolution—deciding whether records such as names, emails and addresses map to the same real-world identity. Hightouch compared its Jev decision model, GPT-6 Luna and Claude Opus, then built a hybrid evaluator in which Jev could ask Opus for help.
The reported result
According to Hightouch, the hybrid approached Opus-level accuracy at more than four times lower cost. Out of the box, the decision models achieved 88.8–97.3% accuracy across the company’s tests, while trailing Opus by about six percentage points on the harder benchmark. The company also reports decision-model costs of $0.05–$0.10 per 1,000 identities and responses around 0.15 seconds.
Confidence quality matters more than a single average
The routing worked because Jev’s mistakes tended to arrive with low confidence. Luna performed better initially, but more of its errors came with high confidence, making them harder to catch. On harder cases, the initial accuracy gap between the decision models narrowed.
That distinction is useful well beyond identity resolution. A production router needs a score that correlates with the probability of being wrong. Without calibration, escalation becomes either wasteful—too many easy cases go to the expensive model—or unreliable.
| Approach | Best role | Main trade-off |
|---|---|---|
| Decision model | High-volume, routine judgements | Lower accuracy on difficult cases |
| Frontier model | Complex or ambiguous judgements | Higher cost and latency |
| Hybrid evaluator | Mixed workloads with useful confidence | Thresholding and evaluation complexity |
How to evaluate the pattern yourself
- Build a hard-case set. Aggregate accuracy can hide the failures that escalation is meant to catch.
- Measure calibration. Plot confidence against observed error instead of treating the score as self-explanatory.
- Price the whole route. Include escalations, retries, latency and the cost of reviewing wrong answers.
- Test threshold policies. The right operating point depends on the cost of a false acceptance versus a false escalation.
These are vendor-reported results on a specific workload, not a universal promise about every judge or routing system. Still, the architecture is worth testing: small models become more useful when the system around them is explicit about uncertainty.
Read Hightouch’s methodology and charts, then compare notes in the .
Discover more from Sulat — AI & Tech News
Subscribe to get the latest posts sent to your email.
