https://hiren.me/tags/h100/ Recent content in H100 on Hugo en Thu, 24 Sep 2026 15:54:50 -0700 https://hiren.me/posts/watching-a-kv-cache-grow-part-3/ Thu, 24 Sep 2026 15:54:50 -0700 https://hiren.me/posts/watching-a-kv-cache-grow-part-3/ <p><a href="https://hiren.me/posts/watching-a-kv-cache-grow/" >Part 1</a> showed what a KV cache is as bytes: 23,040 per token for SmolLM2-135M. <a href="https://hiren.me/posts/watching-a-kv-cache-grow-part-2/" >Part 2</a> showed that decode gets slower as the cache grows, because every step has to move it through memory.</p> <p>Where I&rsquo;m heading with this series is disaggregated inference. The idea is to run prefill on one set of GPUs and decode on another, because the two phases want different things (parts 1 and 2 are really about that difference). The catch is that the cache prefill builds has to get from the prefill machine to the decode machine before decoding can start.</p>