Inference disaggregation (part 2): moving the cache

Fri, 25 Sep 2026 11:10:00 -0700

Part 1 showed why you’d split prefill and decode onto different GPUs: a long prefill on the same GPU makes everyone who is already decoding there wait. Splitting has a price, though. The KV cache that prefill builds has to move to the GPU that does decode. The offload post (part 3 of Watching a KV cache grow) predicted that moving a 32k-token cache would add about 3% to prefill over a 400 Gb/s link and about 12% over 100 Gb/s. This post measures the fastest case, within one machine, on real hardware.

Inference disaggregation (part 1): why split prefill and decode

Fri, 25 Sep 2026 11:05:00 -0700

In Watching a KV cache grow I looked at the two phases of LLM inference: prefill, which processes a whole prompt in one go, and decode, which produces one token at a time. They behave very differently. Prefill is one big burst of work that scales with the prompt. Decode is a long run of small steps, each one reading the whole KV cache.

Disaggregated inference splits the work of serving a model across separate hardware. The most common form, and the one this series starts with, runs prefill and decode on different GPUs. Other forms split each layer’s attention and feed-forward parts onto different chips, or keep the KV cache in its own memory pool shared by many GPUs.

Disaggregation on

Inference disaggregation (part 2): moving the cache

Inference disaggregation (part 1): why split prefill and decode