LLM on

Watching a KV cache grow (part 3): move it or rebuild it?

Thu, 24 Sep 2026 15:54:50 -0700

Part 1 showed what a KV cache is as bytes: 23,040 per token for SmolLM2-135M. Part 2 showed that decode gets slower as the cache grows, because every step has to move it through memory.

Where I’m heading with this series is disaggregated inference. The idea is to run prefill on one set of GPUs and decode on another, because the two phases want different things (parts 1 and 2 are really about that difference). The catch is that the cache prefill builds has to get from the prefill machine to the decode machine before decoding can start.

Watching a KV cache grow (part 2)

Thu, 24 Sep 2026 12:20:56 -0700

In part 1 I watched a KV cache fill up one token at a time on my laptop, and worked out that SmolLM2-135M’s cache costs 23,040 bytes per token. The post ended with a question: every decode step has to read the weights and the whole cache, so as the cache grows, does each step get slower in proportion to the extra bytes?

The short answer is yes, in a straight line, but each step moves the cache through memory about 3 times rather than once. With a different cache class it’s closer to 7.

Watching a KV cache grow (part 1)

Thu, 24 Sep 2026 11:12:06 -0700

Glenn Lockwood’s What are KV caches, really? explains prefill, decode, and why a KV cache exists, and then takes apart some vendor speedup claims for KV offload. It’s a good read and it got me curious.

I’m an infra person, not an ML person, so I wanted to see a KV cache for real, the way I’d look at any other buffer: what’s in it, how it grows, and how big it gets.