https://hiren.me/tags/reliability/ Recent content in Reliability on Hugo en Wed, 30 Sep 2026 11:00:00 -0700 https://hiren.me/posts/inference-in-production-part-3/ Wed, 30 Sep 2026 11:00:00 -0700 https://hiren.me/posts/inference-in-production-part-3/ <p><a href="https://hiren.me/posts/inference-in-production-part-1/" >Part 1</a> followed a request through prefill and decode pools, and <a href="https://hiren.me/posts/inference-in-production-part-2/" >part 2</a> covered the gateway, router and autoscaler in front of them. This part is about what happens when a GPU, a node or an NVL72 tray fails under them: how often that happens, how it shows up, how much of the deployment it takes out, and what becomes of the requests that were running on it. The failure data comes from published papers and NVIDIA&rsquo;s docs; the part about requests in flight I measured by killing vLLM replicas on Modal. The script is in the <a href="https://github.com/hirenp/kv-cache-lab" class="external-link" target="_blank" rel="noopener">lab repo</a>.</p>