https://hiren.me/tags/prefix-caching/ Recent content in Prefix Caching on Hugo en Thu, 08 Oct 2026 17:30:00 -0700 https://hiren.me/posts/watching-a-kv-cache-grow-part-4/ Thu, 08 Oct 2026 17:30:00 -0700 https://hiren.me/posts/watching-a-kv-cache-grow-part-4/ <p><a href="https://hiren.me/posts/watching-a-kv-cache-grow-part-3/" >Part 3</a> asked whether a cache that already exists is cheaper to copy back to the GPU or to rebuild, and found copying wins: &ldquo;any link faster than about 2 GB/s (16 Gb/s) beats rebuilding&rdquo;. That was a script moving one cache by hand. This part looks at the same idea inside a serving engine, where it matters every day: a chat app resends the whole conversation on every turn, and an agent resends its context on every call. <a href="https://hiren.me/posts/inference-in-production-part-5/" >Inference in production part 5</a> quoted a measurement of Claude Code: &ldquo;a median Claude step reads back 126k prefix tokens but appends only 857&rdquo;. If the server still has the cache for those 126k tokens, it only has to prefill the 857 new ones.</p>