In Watching a KV cache grow I looked at the two phases of LLM inference: prefill, which processes a whole prompt in one go, and decode, which produces one token at a time. They behave very differently. Prefill is one big burst of work that scales with the prompt. Decode is a long run of small steps, each one reading the whole KV cache.
Disaggregated inference splits the work of serving a model across separate hardware. The most common form, and the one this series starts with, runs prefill and decode on different GPUs. Other forms split each layer’s attention and feed-forward parts onto different chips, or keep the KV cache in its own memory pool shared by many GPUs.
This first post is about why you’d split prefill and decode at all: what happens to someone who is already getting tokens when a long prompt shows up on the same GPU.
The short answer: they stop getting tokens for as long as the new prompt’s prefill takes, which is 1.5 seconds for a 32k-token prompt. Moving the prefill to a second GPU fixes it, but only if it runs in a separate process. A separate thread in the same Python process didn’t help.
Setup Link to heading
Machine: 2x NVIDIA H100 SXM (80GB HBM3), NVLink between them (Modal)
Software: Python 3.11, PyTorch 2.14.0, Transformers 5.17.0
Model: Qwen2.5-7B-Instruct, one copy on each GPU
The script is kv_interference.py in the same repo as the KV cache series, run on Modal through modal_run.py.
The scenario Link to heading
User A sent a 1,024-token prompt a while ago and is now getting tokens back, about one every 15 ms. Just before A’s 20th token, request B arrives with a long prompt: 4k, 16k or 32k tokens. I record the gap between every pair of A’s tokens, and how long B waits for its first token.
Something has to decide what the GPU does next. I modelled it the way, as far as I know, serving engines schedule work: one step at a time, taking turns. Each step either produces one of A’s tokens or does some of B’s prefill. I tried three ways:
- Colocated: run B’s whole prefill on A’s GPU, then carry on with A.
- Chunked prefill: split B’s prompt into chunks (512 or 2,048 tokens) and alternate: one chunk of B, one token for A, one chunk of B, and so on. This is what serving engines call chunked prefill, and vLLM turns it on by default.
- Separate GPU: A decodes on GPU 1 while B’s prefill runs on GPU 0. I did this two ways: B’s prefill in a separate thread of the same Python process, and in a separate process with its own copy of the model.
The chunked loop is the interesting one. Each chunk is a normal forward call that appends to B’s cache:
for step in range(A_STEPS):
if b_pos < len(b_prompt): # one chunk of B's prefill
b_logits, b_cache = prefill(model, b_prompt[:, b_pos:b_pos + chunk], b_cache)
b_pos += chunk
token = logits.argmax().view(1, 1) # then one token for A
out = model(input_ids=token, past_key_values=a_cache, use_cache=True)
In every mode A’s 100 tokens came out exactly the same as when A ran alone. The scheduling changes when A’s steps run, not what they compute. B’s first token was also the same token as from a single unchunked prefill every time. That’s the same token, not necessarily identical scores, since chunked and unchunked prefill take different paths through the GPU.
Results Link to heading
Data table
Switch between prompt lengths and scales. With “same for all rows”, the colocated stall is to scale and the chunked slowdowns are barely visible; with “each row its own”, you can see their shape.
B = 16k tokens B = 32k tokens
mode A's worst gap slow gaps B's TTFT A's worst gap slow gaps B's TTFT
A alone 19 ms 0 - 19 ms 0 -
colocated 578 ms 1 559 ms 1,472 ms 1 1,456 ms
chunked, 512 74 ms 32 1,532 ms 111 ms 63 4,314 ms
chunked, 2,048 188 ms 8 1,013 ms 308 ms 16 2,995 ms
separate GPU, thread 480 ms 2 574 ms 1,161 ms 2 1,464 ms
separate GPU, process 21 ms 0 562 ms 26 ms 0 1,459 ms
“Slow gaps” counts A’s gaps longer than twice A’s normal gap (15 ms). TTFT is B’s time to first token, measured from when B arrives.
Colocated: A’s gaps sit at about 15 ms, then one gap is as long as B’s whole prefill: 578 ms at 16k and 1.5 seconds at 32k. For that stretch A gets nothing. In a real server with many users decoding on the GPU, all of them would stall at once.
Chunked prefill: the one big stall turns into a run of smaller ones. With 512-token chunks at 32k, A’s worst gap drops from 1,472 ms to 111 ms, but 63 of A’s 80 remaining gaps are slow. They grow as B’s prefill goes on, from about 30 ms to 111 ms, because each chunk works over all the chunks before it. B pays too: its first token takes 4.3 seconds instead of 1.5. Bigger chunks sit in between: with 2,048-token chunks, A gets 16 slow gaps of up to 308 ms, and B waits 3 seconds.
Part of that cost comes from how Transformers runs chunks, not from chunking itself. Every chunk after the first passes a mask to attention, and with a mask Transformers stops sharing the KV heads and expands the whole cache before attention, in every layer. It’s the same thing part 2 of the KV cache series found with StaticCache. The cache grows with each chunk, so that copying grows too. Serving engines use their own attention kernels that don’t do this, so the trade-off is real, but the size of B’s delay here is specific to this setup.
Separate GPU, process: A doesn’t notice B at all. Its worst gap at 32k is 26 ms, no gaps are slow, and B gets its first token in 1.46 seconds, the same as its prefill on its own. This is the half of disaggregation that comes free. B’s cache never moved to GPU 1 here; part 2 measures that half.
Why a thread wasn’t enough Link to heading
My first version ran B’s prefill in a thread of the same Python process, and it barely helped: A still stalled for 1.2 seconds at 32k, even though A and B were on different GPUs.
The GPUs weren’t the problem; something inside the one Python process tied the two threads together. I haven’t traced what. Both threads share one interpreter lock, and one possibility is that something in B’s prefill path waits for the GPU while holding that lock, so A’s thread can’t run until B’s work finishes. It could also be something else in running PyTorch and Transformers from Python. What I did measure is that moving B into its own process, with its own interpreter, made the stall disappear completely.
For anyone building this: in this setup, two GPUs weren’t enough on their own. Prefill and decode need to be separate processes (or separate machines), which is how disaggregated serving systems run them anyway.
What this means Link to heading
On one GPU, a long prompt forces a choice between two bad options. Run it all at once and everyone already decoding stalls for as long as the prefill takes. Chunk it and they get a run of smaller stalls, while the new request waits longer for its first token (three times longer here, though part of that is specific to this setup). Chunk size only moves you along that trade-off.
A second GPU running prefill in its own process takes the trade-off away: A keeps getting tokens on schedule and B gets its prefill at full speed. The price is that B’s KV cache has to move to the decode GPU before B can start decoding there. In this experiment B’s cache stayed on GPU 0. Part 2 measures what moving it costs.
With one decoding user, this doesn’t yet show that splitting the phases is what helps. Sending B to an idle second GPU that runs both prefill and decode would have spared A too. The case for splitting shows up under load: when every GPU that runs both phases has users decoding on it, a long prompt stalls someone wherever it lands, while GPUs that only decode never see a prefill. Showing that needs many users at once, which later posts in this series add.
Caveats Link to heading
- One decoding user. A real server batches many users’ decode steps together, and a stall hits all of them at once, but I didn’t model batching.
- The scheduler is a Python loop that takes turns, not a real serving engine. vLLM, for example, puts a prefill chunk into the same batch as the pending decode steps rather than alternating separate calls, and sizes chunks with
max_num_batched_tokens. Its docs describe the same trade-off measured here: smaller chunks are better for the gap between tokens, bigger ones for time to first token. - In the separate-process mode, B’s cache isn’t copied to GPU 1, so B’s time to first token leaves out that copy. Part 2 measures it at about 6 ms for a 32k-token cache over NVLink.
- cuDNN attention was off for the whole run, prefill included, for the reason in the offload post; it’s a global setting, so it can’t be switched per call while the thread mode runs. That makes prefill here about 25% slower than in the offload post and part 2, which left it on for prefill: 1,456 ms at 32k here, against 1,167 to 1,201 ms there.
Next Link to heading
Part 2: what it costs to move the cache from the prefill GPU to the decode GPU.