Inference in production (part 1): disaggregated inference on B200 and GB200 NVL72

Tue, 29 Sep 2026 11:30:00 -0700

This series is about how LLM inference is deployed in production: what the hardware looks like, where each piece of work runs, and how one request moves through it. The numbers in this post come from NVIDIA’s documentation and from what DeepSeek and Moonshot have published about their own systems. Where a piece overlaps with something I measured earlier, I link to it.

The short version: production systems split each request’s two phases across separate pools of GPUs, and the design question that changes the picture most is where the NVLink boundary sits. On an HGX B200 system it’s the node, so the KV cache crosses the network between the pools. On a GB200 NVL72 it’s the whole rack, so the cache can stay on NVLink, as long as the deployment fits inside that rack.

InfiniBand on

Inference in production (part 1): disaggregated inference on B200 and GB200 NVL72