Prompt Caching In Agents

Summary

Large language models are often thought of like functions: send in some text, receive some text. That is a useful abstraction, but it ignores one of the most important parts of running a coding agent: most of the input is the same as last time. In other words we mostly append to it. A coding agent sends the model its system prompt, tool definitions, project instructions, conversation history, tool calls, and tool results. On the next turn it sends almost all of that again, plus a small amount of new material. Once a session has grown to tens or hundreds of thousands of tokens, recomputing the whole prompt for every turn is slow and expensive. Prompt caching is what makes this somewhat economic, but it is also quite fragile. A changed tool definition, a model switch or a provider routing decision can turn what one would expect to be a cheap incremental request into a full replay of the context. For coding agents, cache behavior is therefore not just an implementation detail or optimization. It affects latency, cost, tool design, session design, and even which product features should be made available. What a KV Cache Contains A transformer processes a prompt in two broad phases. During prefill, it reads the input tokens and computes attention state for them. During decode, it produces new tokens one at a time. At each attention layer, every processed token produces a key and a value. These are not quite like key-value lookups in a hash table: both are arrays of numbers, usually floats or lower-precision quantized values. When processing a new token, the model compares that token's query with the earlier keys to determine how relevant each earlier token is. It then uses those relevance scores to form a weighted mixture of the corresponding values. In that sense, a key is what the model matches against, while a value is the information it retrieves (but the lookup is fuzzy rather than "returning a single exact match" like a dictionary lookup.) Those keys and values are...