Kubernetes on

Inference in production (part 4): rolling out a new model

Wed, 30 Sep 2026 12:30:00 -0700

The first three parts followed a request through prefill and decode pools, the router and autoscaler in front of them, and what happens when the hardware under them fails. This part is about changing what runs on all of it: a new model, a new checkpoint of the same model, or a new engine version.

A rollout does to every replica, on purpose, what the last two posts measured happening by accident. Each old replica is shut down, like part 3’s failures, and each new one pays part 2’s cold start. A rollout goes well when users don’t notice either. The sources are project docs and operators’ own write-ups; the part about draining I checked on Modal.

Inference in production (part 3): when hardware goes bad

Wed, 30 Sep 2026 11:00:00 -0700

Part 1 followed a request through prefill and decode pools, and part 2 covered the gateway, router and autoscaler in front of them. This part is about what happens when a GPU, a node or an NVL72 tray fails under them: how often that happens, how it shows up, how much of the deployment it takes out, and what becomes of the requests that were running on it. The failure data comes from published papers and NVIDIA’s docs; the part about requests in flight I measured by killing vLLM replicas on Modal. The script is in the lab repo.

Inference in production (part 2): the gateway, the router and the autoscaler

Wed, 30 Sep 2026 00:40:00 -0700

Part 1 followed one request through prefill and decode pools on B200 and GB200 NVL72. This part covers what sits in front of those pools: the gateway that accepts the request, the router that picks the GPUs for it, and the autoscaler that decides how many GPUs there are. The last section measures why a new replica takes minutes to become useful, with vLLM cold starts I timed on H100s. The rest comes from project docs and published benchmarks, linked where they’re used.