Part 1 showed what a KV cache is as bytes: 23,040 per token for SmolLM2-135M. Part 2 showed that decode gets slower as the cache grows, because every step has to move it through memory.
Where I’m heading with this series is disaggregated inference. The idea is to run prefill on one set of GPUs and decode on another, because the two phases want different things (parts 1 and 2 are really about that difference). The catch is that the cache prefill builds has to get from the prefill machine to the decode machine before decoding can start.
Before getting to that, there’s a simpler question underneath it, the one KV offload and prefix caching depend on: if you already have a cache, is moving it back cheaper than rebuilding it? It’s the trade-off behind the KV offload claims in Glenn Lockwood’s post that started this series. Disaggregation asks a slightly different question, which I get to at the end.
For a 7B model, reloading won from every place I tried, at every length.
Setup Link to heading
My laptop can’t answer this one (host memory and GPU memory are the same memory on a Mac), so this runs on an H100 rented by the second on Modal.
GPU: NVIDIA H100 SXM (80GB HBM3), host link PCIe Gen 5 x16
Container: 8 CPU cores, 64 GB RAM (Modal)
Software: Python 3.11, PyTorch 2.14.0, Transformers 5.17.0
Models: SmolLM2-135M-Instruct (same as parts 1 and 2)
Qwen2.5-7B-Instruct (57,344 bytes of KV per token, measured)
I added a 7B model because this question depends on how expensive prefill is. For a 135M model prefill is nearly free, so a comparison with only SmolLM2 would say little about real systems. The new script is kv_offload.py, and modal_run.py runs it on Modal.
What I measured Link to heading
For each context length, staying within each model’s trained limit this time (1k, 4k, 16k and 32k tokens for Qwen; 1k and 4k for SmolLM2, whose limit is 8k):
- Recompute: time a prefill of that many tokens. That’s the cost of rebuilding the cache from scratch.
- Offload: copy the finished cache out of GPU memory to a tier.
- Reload: copy it back into GPU memory, ready for decode.
The tiers are pinned host RAM, ordinary (pageable) host RAM, a file on local disk, and a file on a Modal Volume, which is Modal’s network storage. I also copied to another buffer on the same GPU, as a ceiling rather than a real place to keep a cache. Each number is the median of 5 runs after a warmup, with a GPU sync around each.
The pinned tier is about as simple as it gets:
host = [torch.empty(t.shape, dtype=t.dtype, pin_memory=True) for t in gpu_tensors]
for h, g in zip(host, gpu_tensors):
h.copy_(g, non_blocking=True) # offload
for g, h in zip(back, host):
g.copy_(h, non_blocking=True) # reload
Every reloaded cache also has to be the same bytes. For each tier the script checks the tensors are equal, then decodes 20 tokens from the reloaded cache and from the original, and checks that the tokens and logits match exactly. That’s the test I suggested in part 1 for offload systems: compare against a normal cached run. All of them passed.
Results Link to heading
Data table
Drag the slider to change the context length. Bars are scaled to the recompute time.
At 32k tokens the cache is 1.8 GiB. Rebuilding it takes 1.2 seconds. Reloading it takes 68 ms from pinned host RAM, 238 ms from pageable RAM, 403 ms from local disk and 563 ms from the Modal Volume.
reload beats recompute by (32k tokens)
GPU -> GPU (ceiling) 858x
pinned host RAM 18x
pageable host RAM 5x
local disk 3x
Modal Volume 2x
The gap also widens with length. Prefill gets a little more expensive per token as the context grows (about 29 µs per token at 1k, 37 µs at 32k), while a reload costs the same per byte at any length.
The break-even link speed Link to heading
One division ties all of this together: bytes of cache per token, divided by prefill time per token. That’s the link speed at which moving a cache and rebuilding it take the same time.
Qwen2.5-7B 1k tokens: 57,344 bytes / 29 µs ≈ 2.0 GB/s
Qwen2.5-7B 32k tokens: 57,344 bytes / 37 µs ≈ 1.6 GB/s
SmolLM2 4k tokens: 23,040 bytes / 4.2 µs ≈ 5.5 GB/s
For Qwen on this H100, any link faster than about 2 GB/s (16 Gb/s) beats rebuilding, and every tier I tried is faster than that.
SmolLM2 goes the other way. Its prefill takes about 17 ms at both 1k and 4k, mostly fixed overhead, the same thing part 2 ran into, so per token it’s very cheap and the break-even is much higher. At 4k tokens, reloading its 90 MiB cache from disk (19 ms, about 5 GB/s) or the Volume (33 ms) is already slower than recomputing it. Whether offload pays off depends on the model, and for small models it often doesn’t.
Other things that showed up Link to heading
Pinned vs pageable is a 3.5× difference. Same data, same PCIe link: about 27 GB/s from pinned memory, about 8 GB/s from ordinary memory. The only difference is how the host buffer was allocated. As far as I know, with pageable memory the driver has to copy everything through a pinned staging buffer first.
Writing is the slow side, but it usually isn’t in the way. Offloading the 32k cache to disk takes 1.2 seconds (write plus fsync), as long as recomputing it. But a cache doesn’t need to survive a crash, so a real system can skip the fsync and write in the background while the conversation is idle. For a conversation that gets evicted and comes back once, the write costs disk bandwidth rather than waiting time, and the reload (403 ms) still beats recomputing (1,201 ms). A reload also runs on the GPU’s copy hardware and leaves its compute free for other requests, where a recompute spends 1.2 seconds of H100 compute.
What this means for disaggregation Link to heading
In a disaggregated setup the question is a bit different. Prefill runs on the prefill machine either way, so the decode machine isn’t choosing between receiving the cache and rebuilding it. What matters is how much the transfer adds to the time before the first token, compared with running prefill and decode on the same machine.
I only have one GPU here, so this is a prediction, not a measurement: the measured cache size divided by the nominal link speed.
Qwen2.5-7B, 32k tokens, 1.8 GiB of cache
prefill (measured): 1,201 ms
+ transfer over 100 Gb/s: ~150 ms (+12%, predicted)
+ transfer over 400 Gb/s: ~38 ms (+3%, predicted)
Those are worst cases. As far as I know, real systems send the cache layer by layer while prefill is still computing later layers, so much of the transfer hides behind prefill. H100 servers also usually have one 400 Gb/s network card per GPU, so 400 Gb/s is the normal case, not the optimistic one.
Disaggregation bets that a few percent on the first token costs less than having prefill and decode compete for the same GPU. Measuring that side is the next post.
This is the prefill/decode kind of disaggregation, where the whole cache moves once per request. There’s also a newer kind that splits each layer’s attention and feed-forward parts onto different hardware (this paper explores it, and NVIDIA’s Groq 3 LPX pairs GPUs with LPUs this way). There the cache stays on the GPU that holds it, and only small intermediate results cross the link on every token.
Caveats Link to heading
Modal gives you a GPU, not a machine you can inspect, and three numbers show it:
- Host-memory bandwidth varied between machines. The pinned reload ran at 27 GB/s in the full run and 47 GB/s in an earlier test, on the same kind of PCIe Gen 5 x16 link, which can carry about 64 GB/s in theory. As far as I know, the usual suspect is which CPU socket the pinned buffer lands on (NUMA placement), and I can’t see or control that from inside the container.
- “Cold” disk reads were probably warm. The script asks the kernel to drop the file from the page cache before each read, but cold and warm times came out the same (403 vs 397 ms). 403 ms also matches a pageable copy to the GPU (238 ms) plus copying the file out of RAM (about 165 ms), which is what a warm read looks like. As far as I know, Modal runs containers under gVisor, a sandboxed kernel, which would explain the page-cache drop not working.
- The Volume’s ~3 GB/s is end-to-end reload throughput, including the copy to the GPU. I can’t confirm the reads crossed the network, and they ran right after a warmup read of the same file, so they may have been served from a local cache.
None of this can flip the host-RAM results, where the margins are 5× to 18×. It could flip the disk and Volume results: their margins are 2× to 3×, and genuinely cold storage might lose.
Next Link to heading
The next post is the other side of that bet: moving a cache between two GPUs and decoding on the second one, first within one box, then between two machines. That’s where a new series on inference disaggregation starts. It needs hardware I can see into rather than a rented container, so it depends on getting access to some.