https://hiren.me/tags/latency/ Recent content in Latency on Hugo en Sat, 03 Oct 2026 12:00:00 -0700 https://hiren.me/posts/inference-in-production-part-5/ Sat, 03 Oct 2026 12:00:00 -0700 https://hiren.me/posts/inference-in-production-part-5/ <p>The first four parts covered <a href="https://hiren.me/posts/inference-in-production-part-1/" >prefill and decode pools</a>, <a href="https://hiren.me/posts/inference-in-production-part-2/" >the gateway, router and autoscaler</a>, <a href="https://hiren.me/posts/inference-in-production-part-3/" >hardware failures</a> and <a href="https://hiren.me/posts/inference-in-production-part-4/" >rollouts</a>. They mostly assumed one kind of traffic, a user chatting. A coding agent, a reasoning model and an overnight batch job send very different requests, and some of the machinery from those posts matters less for them, or more.</p> <p>This part describes a workload with a few numbers, goes through the common types, and then goes back through parts 1 to 4 to see which workloads each piece is for. The last section measures, on one H100, how much a few long prompts slow down everyone else&rsquo;s answers, and which latency numbers show it.</p>