https://hiren.me/tags/vllm/ Recent content in VLLM on Hugo en Wed, 30 Sep 2026 00:40:00 -0700 https://hiren.me/posts/inference-in-production-part-2/ Wed, 30 Sep 2026 00:40:00 -0700 https://hiren.me/posts/inference-in-production-part-2/ <p><a href="https://hiren.me/posts/inference-in-production-part-1/" >Part 1</a> followed one request through prefill and decode pools on B200 and GB200 NVL72. This part covers what sits in front of those pools: the gateway that accepts the request, the router that picks the GPUs for it, and the autoscaler that decides how many GPUs there are. The last section measures why a new replica takes minutes to become useful, with vLLM cold starts I timed on H100s. The rest comes from project docs and published benchmarks, linked where they&rsquo;re used.</p>