Sulat.com / Writing on AITuesday, November 17, 2026
Agent Engineering/17 November 2026

Together AI and Ollama explain how to keep prefix caches intact in long coding-agent sessions

Zain Hasan of Together AI and Parth Sareen of Ollama say the harness and the inference engine each have to protect the prefix cache. Their post gives no measured hit rates.

Glossy 3D Together AI logo in front of a stack of glowing glass slabs and server racks, standing for a cached prompt

At a glance

  • Zain Hasan of Together AI and Parth Sareen of Ollama say a long coding-agent session stays fast only if both the harness and the inference engine protect the prefix cache.
  • They say a prefix cache matches from the start of the prompt to the first differing token, so a change near the top forces the engine to reprocess the rest of the session.
  • On the harness side, they name late-loaded tools, changing system-prompt values, mid-task reasoning-effort changes, oversized tool results and subagent setup as the habits that break the cache.
  • On the engine side, they say requests must reach the node holding the cache, and idle caches can be moved to CPU memory between turns.
  • The post gives no measured hit rates or cost savings; its 55-against-10 and 5,050-against-100 figures are the authors' own idealised arithmetic.

Together AI and Ollama have published a joint post on keeping the prefix cache intact in long coding-agent sessions. Zain Hasan of Together and Parth Sareen of Ollama say a session only stays fast at turn 90 if two parties both do their part: the harness, the agent software that builds each request, and the inference engine, the software that runs the model on GPUs.

Together published the post on 16 November, US time. The authors say Ollama works on harness integrations, which means deciding what goes into each request, and Together works on the engine, which means deciding what happens when that request arrives. On the Ollama side, they say, the ollama launch command connects harnesses such as Claude Code, Codex and Pi to open models such as DeepSeek-V4.1-Flash and GLM-5.3.

Why long sessions lean on the cache

A coding agent resends the whole conversation on every step: the system prompt, every tool definition, the request, each file it has read and each command it has run. What comes back is usually a tool call a few dozen tokens long. Hasan and Sareen call this a large input and a small output, and say it’s typical of agentic workloads. Prefix caching, in which the engine keeps the work it did on earlier tokens and reuses it across requests, stops the GPU redoing that work on each turn.

Their arithmetic assumes every turn adds the same amount. Ten turns without caching process 1 + 2 + … + 10 = 55 units of prompt, and with caching they process 10. At 100 turns it’s 5,050 against 100. The authors call these idealised numbers, because real turns vary in size and every new token still has to attend to the cached ones. Without caching, they say, the work redone grows with the square of the session length.

Bar chart: ten turns process 55 prompt units without a cache and 10 with one; 100 turns process 5,050 against 100
Over 100 turns, caching cuts 5,050 units of prompt work to 100 in the authors’ idealised sums.

The catch is how matching works. A prefix cache matches from the start of the prompt up to the first token that differs, and everything after that point is processed again. Adding to the end costs nothing extra. Changing a single character near the top means the engine reprocesses the rest of the session.

Five stacked prompt layers: the system prompt is reused, while the edited tool list and every layer below it are read again
One edit near the top of the prompt makes the engine read everything below it again.

What the harness has to get right

The authors list five habits that break the cache, and the table below sums up each one. Most of them come down to where in the prompt a change lands.

Harness habitWhat it costsWhat the post says to do
Adding a tool mid-sessionThe tool list sits near the top, so everything after it is processed againAppend the definition at the end of the conversation, where new content goes anyway
Timestamp, git status or working directory in the system promptEvery turn is a full re-readPut it in the task message once and keep the system prompt identical
Raising reasoning effort mid-taskIn GLM-5.3-Flash the chat template writes effort into the first system blockKeep effort where it started, as in the post’s worked example
Large tool resultsEvery remaining turn pays for them, and they use up context window even when cachedCut the output down, and take care how you cut it
Starting a subagentA subagent with its own system prompt shares nothing with the parentInherit the parent session, or send a brief with the same system prompt and tools
Harness habits that break the prefix cache, as set out in the Together AI and Ollama post.

Tools are the first case. Harnesses now carry many of them, so newer ones load a tool only when the agent searches for it. Adding the definition to the tool list works, the authors say, but on a long session it can mean hundreds of thousands of tokens processed again. Appending it at the end avoids that, except that some chat templates don’t expect a tool definition halfway through a conversation and the call errors out. Whether a harness can load tools late, they say, depends on the model.

Reasoning effort is the one Ollama hit itself. Raise it from low to high on a stalled task and the next request differs from the last at almost the first token, so the whole session is processed again. Facts that stay fixed within a session but differ between sessions, such as the user’s name or the start date, don’t hurt that session, the authors say. They do stop a new session reusing a setup that’s already cached.

Subagents have a trade-off. A subagent that inherits the parent’s system prompt, tools and conversation starts with the whole prefix cached. The other option is a brief: the same system prompt and tools, then only the files and instructions the subagent needs. Inheriting isn’t strictly better, the authors say, because a cached prefix is cheap to process but not free. Every token the subagent generates still reads through it, and it occupies GPU memory the whole time.

What the engine has to get right

Together says a session’s cache lives on a particular node in a particular data centre. If the next turn lands elsewhere, that node processes every earlier turn from scratch and two nodes end up holding the same prefix. The post describes two ways to pick the node. One is a key such as prompt_cache_key that the client sends with every request, so requests with the same key go to the same node. The other hashes the prefix and sends the request to whichever node has the longest matching prefix cached, even if another session put it there.

The second method is why an identical system prompt matters. If every session of a harness starts with the same system prompt and tool definitions, a brand-new session can find that setup already cached. Unique identifiers in the prompt guarantee a cold start. Routing also has to account for load. When the warm node is overloaded, the post says, taking an idle node and paying to process the prefix again can be the better deal.

Two checklists: the harness keeps the prompt stable, and the engine routes turns to the warm node and keeps caches alive
The harness protects the start of the prompt, and the engine keeps the cache within reach.

Keeping the cache alive between turns is the other job. An agent often makes a tool call and then waits a couple of minutes while tests run. Holding that cache in GPU memory takes room active users need, and discarding it means re-reading everything when the agent returns. Together says it offloads the cache to CPU memory while the session is idle and restores it for the next turn. It also hands out cache memory in small blocks, a technique called PagedAttention, and rebuilds the running batch on every pass, called continuous batching, so five subagents fired at once each take a slot as one frees up.

The authors say headroom for cache memory, not compute, usually limits how many requests a GPU can serve at once. Together also runs prefill, which reads the prompt, and decode, which writes the answer, on separate clusters, with the cache shipped between them over RDMA, a fast direct memory link. It says a single GPU costs around US$50,000 a year to rent, so it shouldn’t sit idle.

The post gives no measured cache hit rates, latency figures or cost savings for Together’s or Ollama’s setups. The 55-against-10 and 5,050-against-100 figures are the authors’ own idealised arithmetic. Anthropic’s API team described later agent requests as mostly appends in a video on 12 November.


Discover more from Sulat AI & Tech News

Subscribe to get the latest posts sent to your email.