Sulat.com / Writing on AISaturday, November 21, 2026
Artificial Intelligence/21 November 2026

Orosz says cheap KV cache reads make coding inference profitable; the filings back part of it

Gergely Orosz says reading from the KV cache is cheap and speculative decoding speeds up coding sessions. Anthropic's S-1 and price list back part of that; no source shows the rest.

A glossy terracotta 3D Anthropic logo beside a stack of frosted-glass memory blocks with glowing orange tokens floating above

At a glance

  • Gergely Orosz said on 20 November that reading from the KV cache is very cheap and that speculative decoding makes sessions more efficient, and that coding use makes heavy use of both.
  • Vlad Mihalcea's usage screen shows 16.3 billion tokens, about 98% of them cache reads; priced at Opus 5.5 list rates it comes to about US$6,500 to US$7,400 with caching and about US$66,600 without, against his US$100,000 figure.
  • Anthropic's S-1 reports US$2.38 billion of cost of revenue on US$16.286 billion of revenue for the first half of 2026, but it doesn't separate cache reads or say how much of the cost is coding workloads.
  • SemiAnalysis, assuming 92% API gross margins, puts a US$200 Claude plan used to its limits on Opus 5.5 at a gross margin of minus 369%, and at an assumed 20% utilisation at 6%.
  • None of the cited sources says Anthropic uses speculative decoding; they describe the technique and open-weight serving stacks such as vLLM and Fireworks.

Gergely Orosz, who writes The Pragmatic Engineer newsletter, said on X on 20 November that reading from the KV cache is “very cheap”, that speculative decoding makes sessions more efficient, and that coding use makes heavy use of both. “THIS is why inference is SUPER profitable!!” he wrote.

He was responding to developer Vlad Mihalcea, who posted a usage screen showing 16.3 billion tokens on what he called his US$200-a-year Pro plan and said the per-use price would be about US$100,000. Priced at Anthropic’s published Opus 5.5 rates, the same screen comes to about US$6,500 to US$7,400 with caching and about US$66,600 without it.

Orosz followed with a second post pointing to two sources: an explainer by Diogo Almeida, and his own longer article with Ivan Klaric, published on 17 November. He added that coding is “such an interestingly amazing fit for inference” and that vendors “won’t want you to know”. The pieces below trace each part of the claim to what’s published.

What Mihalcea’s usage screen shows

The screenshot lists 487.4 thousand input tokens, 88.6 million output tokens, 15.9 billion cache read tokens and 301.8 million cache write tokens across 63 sessions, with Opus 5.5 as his favourite model. Cache reads are about 98% of the total. The screen doesn’t split tokens by model, so the table prices all of them at Opus 5.5 rates from Anthropic’s prompt caching page.

Token typeTokensOpus 5.5 rate per millionCost
Cache reads15.9 billionUS$0.20US$3,180
Output88.6 millionUS$20US$1,772
Cache writes (5-minute)301.8 millionUS$5US$1,509
Fresh input487.4 thousandUS$4US$2
TotalUS$6,463
Token counts from the usage screen Mihalcea posted; rates from Anthropic’s prompt caching page. With 1-hour cache writes at US$8 per million, the total is US$7,368.
Hand-drawn bar chart: a claimed US$100,000 token bill against about US$66,600 with no caching and about US$6,500 to US$7,400 with caching, with the cached bill split into reads, output and writes
Sulat calculation from Mihalcea’s usage screen at Anthropic’s list rates for Opus 5.5.

Reads are the largest line at about half the cached bill, ahead of output. The US$100,000 figure doesn’t come out of the screenshot even with caching removed entirely: all 16.2 billion input-side tokens at the full US$4 per million rate come to US$64,809, and output adds US$1,772. Mihalcea’s post doesn’t show the calculation behind US$100,000, and a list price isn’t Anthropic’s cost to serve, which is the point Orosz is making.

Why an agent session is mostly cache reads

Almeida, the chief executive of TypeSafe AI, wrote the explainer Orosz recommends, “(KV) Cache Rules Everything Around Me“, on 9 September. An agent is a loop: the model reads its context, emits a tool call, the tool returns some tokens and the whole thing goes back in. With a context of N tokens and T tool calls, Almeida says, the model processes roughly N times T input tokens to produce a small number of output tokens, so the cost of a session grows with the square of its length.

Martin Alderson put numbers on that on 10 August. He modelled an agent session that starts at 60,000 tokens of context, with each turn adding 500 tokens of output and 5,000 tokens of tool results, and assumed every request hit the cache. After 20 turns, cache reads were 44.9% of a US$2.32 Opus 5 bill. After 100 turns they were 76.4% of US$21.34, with fresh input at 17.7% and output at 5.9%. Cutting 10% of the turns cut the cost of a run by about 16%, he says.

Hand-drawn chart of five agent turns, each bar taller than the last, plus a bar showing cache reads at 76 percent of a 100-turn session's bill
Each turn re-reads everything before it. Source: Martin Alderson’s session model, 10 August 2026.

The KV cache is the saved result of the model’s reading. For every token it processes, the model computes a set of key and value vectors, and it keeps them so it doesn’t have to redo the work; the prefill and decode explainer covers the two phases. A cache write is a prefill whose results the provider keeps, and a cache read reuses them with no new compute for those tokens.

Almeida’s argument for why reads cost the provider little rests on how decoding works. To write each output token, the model reads the whole cache out of GPU memory, which he describes as very little compute and an enormous amount of memory traffic. A pause for a tool call keeps those bytes in memory and uses neither compute nor memory bandwidth. His conclusion is that most of the cost of cache reads is already covered by the price of output tokens. That’s Almeida’s reasoning, not a figure from Anthropic.

What the price list and rate limits say

Anthropic prices a cache hit at 0.1 times the base input price by default, 0.05 times for Opus 5.5 and Sonnet 5.5, and 0.025 times for Fable 5.1 and 5.2. A cache write costs 1.25 times the base price for the 5-minute lifetime and 2 times for one hour. On Opus 5.5 that’s US$4 per million tokens for input, US$0.20 for a read, US$5 for a 5-minute write and US$20 for output.

Cache reads are also excluded from the input-token rate limits. Anthropic’s rate limit page says the default limits count only uncached input tokens, and that at an 80% hit rate a 2 million token-per-minute limit covers about 10 million total input tokens. Almeida reads that as Anthropic telling developers what a cache read costs it, since limits exist to protect capacity. Anthropic’s page doesn’t say that.

The price of reads has dropped with each recent model. SemiAnalysis says Fable 5.1 cut cache reads by 75% against Fable 5, and Opus 5.5 cut them by 60% against Opus 5 while taking 20% off input and output. Anthropic’s table lists Fable 5.2 at US$0.20 per million, down from US$0.25 on Fable 5.1. Almeida expects the price to keep falling.

Alderson tried to price the other side of the trade. In his 100-turn run, the US$16.31 of reads covers about 3.3 hours of holding roughly 10 GB of cache, assuming 30 KB per token. That works out near US$0.50 per GB-hour, against well under a US cent for memory on AWS, so he calls it probably a huge profit centre. The 30 KB per token is his assumption.

What speculative decoding adds

A normal decode step runs the big model once per output token. Speculative decoding has a cheaper drafter guess several tokens ahead, then the big model checks all of them in a single pass and keeps the ones it agrees with. Leviathan, Kalman and Matias at Google reported a 2x to 3x speed-up on T5-XXL at ICML 2023, with identical outputs. NVIDIA’s engineering blog describes the same draft-then-verify flow.

Hand-drawn comparison: normal decoding takes one pass per token, while speculative decoding drafts five tokens and checks them in a single pass, accepting four
An illustrative draft-then-verify step. Sources: Leviathan et al. (ICML 2023), NVIDIA, vLLM documentation.

Coding suits it because code repeats itself. Fireworks’ documentation recommends n-gram speculation, which matches against the prompt and earlier output instead of running a second model, for “repetitive output, code editing, and structured generation where output often repeats the prompt or prior context”. The vLLM documentation lists EAGLE, multi-token prediction, draft-model, n-gram and suffix methods, and says they target memory-bound workloads at low to medium request rates. N-gram and suffix methods give low to medium gains but need no extra model. Fireworks warns that a poorly matched drafter can make generation slower. Tencent said this week that its DFly2 drafter reuses the main model’s KV cache.

None of the sources cited here says Anthropic uses speculative decoding, or how much it saves if it does. They describe the technique and the open-weight serving stacks that document it.

What Anthropic’s S-1 shows

Anthropic’s S-1, filed on 26 October, reports revenue of US$16.286 billion for the first half of 2026 and cost of revenue of US$2.38 billion, or 14.6% of revenue. A year earlier the figures were US$1.163 billion and US$1.019 billion, or 87.6%. Anthropic says substantially all of cost of revenue is compute and infrastructure for customer-serving workloads, plus certain personnel and overhead costs.

Total compute and infrastructure expense was US$8.945 billion, so most of the compute bill sits in other lines; research and development alone was US$9.741 billion. The filing doesn’t label a gross margin. Subtracting cost of revenue from revenue gives about 85%, which is arithmetic, not a figure Anthropic reports. The operating loss for the half was US$693 million.

Consumption-based revenue was US$13.1 billion, or 80% of the total, and the remainder was subscription revenue. Nothing in the filing separates the cost of cache reads from other serving costs, breaks out coding workloads, or tests Orosz’s claim directly.

Where heavy subscribers fit

SemiAnalysis tested Anthropic’s plans in June and said on 23 August that a US$200-a-month Claude plan can yield up to US$8,000 a month of API-equivalent tokens if the weekly limits are exhausted on long-horizon tasks. Its newsletter method converts the share of a plan’s limit that a million tokens of each type consumes into dollars. At list price Almeida calls that a subsidy of 40 times or more.

SemiAnalysis also modelled margins, assuming 92% gross margins on the API. On that assumption, a plan used to its limits on Opus 5.5 runs at a gross margin of minus 369%, and at an assumed 20% average utilisation at 6%. Roman Leventov asked Orosz whether the US$8,000 estimate already counted cache reads, and Marco Hefti replied that he’d excluded cache reads and got about the same numbers. Orosz’s claim is about the cost of serving tokens and SemiAnalysis’s figures are about plan pricing, so the two answer different questions.

What it changes for people building agents

Almeida’s advice is to measure cache hit rate and the number of tool calls, because the number of calls matters more than the tokens in each. He replayed Nvidia’s Open-SWE-Traces dataset at Fable 5 prices: US$1.07 million at a 100% hit rate, and US$1.97 million at the 89% hit rate reported for Claude Code, which is 84% more.

He also argues that routing individual steps to a cheaper model doesn’t save money for agents, because a KV cache isn’t shared between models and the second model has to write the context again. In his example, a 100,000-token Opus 5 session whose next step generates 1,000 tokens and gets 3,000 back from a tool cost almost three times more when routed to Sonnet and back. Perplexity’s Denis Yarats made a similar argument about routing on 19 November.

For teams running agents at scale, Almeida says self-hosting an open-weight model keeps the cache savings with the operator instead of the lab. Orosz and Klaric’s article, which is paid, covers when to leave lab APIs. Its free section cites Uber cutting per-token costs by about 50%, Pinterest by 90% and AT&T by 56% after moving to open models, and lists speculative decoding as a step once dedicated GPUs have spare compute during decode.

Anthropic’s price list is the only part of this chain that’s public: a read costs US$0.20 per million tokens on Opus 5.5 and Fable 5.2, and what it costs Anthropic to serve one isn’t published.


Discover more from Sulat AI & Tech News

Subscribe to get the latest posts sent to your email.