Everything inference: an intro

Follow one request, "The capital of France is", from the user to the GPU and back. Each stop explains one part of serving an LLM, lets you try it, and links to the posts that go deeper.

This is an introduction: each stop explains one idea with one example, and many techniques production systems use are left out.

Scroll to follow the request ↓

The user The capital of France is The front door Gateway auth, rate limits Router picks a replica Autoscaler adds replicas For comparison: one GB200 NVL72 rack One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. One GPU. Each column is a compute tray of 4. NVLink switch tray: connects all 72 GPUs to each other. NVLink switch tray: connects all 72 GPUs to each other. NVLink switch tray: connects all 72 GPUs to each other. NVLink switch tray: connects all 72 GPUs to each other. NVLink switch tray: connects all 72 GPUs to each other. NVLink switch tray: connects all 72 GPUs to each other. NVLink switch tray: connects all 72 GPUs to each other. NVLink switch tray: connects all 72 GPUs to each other. NVLink switch tray: connects all 72 GPUs to each other. 18 trays × 4 GPUs, plus 9 NVLink switch trays The whole rack is one NVLink domain: 72 GPUs The datacenter An HGX B200 node, 8 GPUs each GPU runs its own replica of Qwen3-8B GPU 0's engine (vLLM or SGLang); every replica runs its own API serverAPI server: takes the request over HTTP and streams tokens back. SchedulerScheduler: picks which requests run together in each step. KV cache managerKV cache manager: hands out blocks of GPU memory to each request's KV cache and frees them when it finishes. Model runnerModel runner: launches the kernels that run the model on the GPU. each dot is one kernel launch GPU 0 · B200 GPU 0 · B200 Compute: the GPU's streaming multiprocessors (SMs), which run the model's kernels. The squares are illustrative, not one per SM. At 2,250 TFLOPS and 7.7 TB/s, the GPU can do about 292 operations for each byte it reads from memory. One user's decode in BF16 does 1 (2 operations per 2-byte weight), so the compute units mostly wait on memory. compute, 2,250 TFLOPS (BF16) memory reads, every step 4 guesses, 1 step no guesses: 1 token per step 3 kept; the model fixes the 4th 1 kept; the model fixes the 2nd HBM, 180 GB, 7.7 TB/sHBM: the GPU's own high-bandwidth memory, 180 GB that it reads at up to 7.7 TB/s. It holds the weights and the KV cache. Weights: the model's trained parameters. Every decode step reads all of them. weights BF16: 16 bits per weight FP8: 8 bits; freed room for more KV cache FP4 weights: 4 bits; freed room for more KV cache KV cache: the keys and values saved for every token in the conversation so far, so they aren't recomputed. It grows by one token per step. +1 token per step KV cache KV cache: the keys and values saved for every token in the conversation so far, so they aren't recomputed. It grows by one token per step. B's 32k prompt A's tokens wait GPU 1 · B200 Weights: the model's trained parameters. Every decode step reads all of them. KV cache: the keys and values saved for every token in the conversation so far, so they aren't recomputed. It grows by one token per step. GPU 2 · B200 Weights: the model's trained parameters. Every decode step reads all of them. KV cache: the keys and values saved for every token in the conversation so far, so they aren't recomputed. It grows by one token per step. GPU 3 · B200 Weights: the model's trained parameters. Every decode step reads all of them. KV cache: the keys and values saved for every token in the conversation so far, so they aren't recomputed. It grows by one token per step. GPU 4 · B200 Weights: the model's trained parameters. Every decode step reads all of them. KV cache: the keys and values saved for every token in the conversation so far, so they aren't recomputed. It grows by one token per step. GPU 5 · B200 Weights: the model's trained parameters. Every decode step reads all of them. KV cache: the keys and values saved for every token in the conversation so far, so they aren't recomputed. It grows by one token per step. GPU 6 · B200 Weights: the model's trained parameters. Every decode step reads all of them. KV cache: the keys and values saved for every token in the conversation so far, so they aren't recomputed. It grows by one token per step. GPU 7 · B200 Weights: the model's trained parameters. Every decode step reads all of them. KV cache: the keys and values saved for every token in the conversation so far, so they aren't recomputed. It grows by one token per step. NVLink: the direct GPU-to-GPU link inside this node, through NVSwitch chips. A KV cache handed from GPU 0 to GPU 1 takes this path. On paper that's 900 GB/s each way per GPU (the 1.8 TB/s counts both directions), 18 times one NIC's 50 GB/s each way. NVLink through NVSwitch: every GPU reaches every other at 1.8 TB/s (both directions) NIC: the network card that connects one GPU to the InfiniBand fabric. NIC 0 NIC: the network card that connects one GPU to the InfiniBand fabric. NIC 1 NIC: the network card that connects one GPU to the InfiniBand fabric. NIC 2 NIC: the network card that connects one GPU to the InfiniBand fabric. NIC 3 NIC: the network card that connects one GPU to the InfiniBand fabric. NIC 4 NIC: the network card that connects one GPU to the InfiniBand fabric. NIC 5 NIC: the network card that connects one GPU to the InfiniBand fabric. NIC 6 NIC: the network card that connects one GPU to the InfiniBand fabric. NIC 7 NVLink domain: these 8 GPUs, prefill on GPU 0 and decode on GPU 1 blue v1, grey restarting, green v2: step through the waves in the widget Another HGX B200 node Engines, one per replica GPU 0 · B200 Weights: the model's trained parameters. Every decode step reads all of them. KV cache: the keys and values saved for every token in the conversation so far, so they aren't recomputed. It grows by one token per step. new replica starting GPU 1 · B200 Weights: the model's trained parameters. Every decode step reads all of them. KV cache: the keys and values saved for every token in the conversation so far, so they aren't recomputed. It grows by one token per step. GPU 2 · B200 Weights: the model's trained parameters. Every decode step reads all of them. KV cache: the keys and values saved for every token in the conversation so far, so they aren't recomputed. It grows by one token per step. GPU 3 · B200 Weights: the model's trained parameters. Every decode step reads all of them. KV cache: the keys and values saved for every token in the conversation so far, so they aren't recomputed. It grows by one token per step. GPU 4 · B200 Weights: the model's trained parameters. Every decode step reads all of them. KV cache: the keys and values saved for every token in the conversation so far, so they aren't recomputed. It grows by one token per step. GPU 5 · B200 Weights: the model's trained parameters. Every decode step reads all of them. KV cache: the keys and values saved for every token in the conversation so far, so they aren't recomputed. It grows by one token per step. GPU 6 · B200 Weights: the model's trained parameters. Every decode step reads all of them. KV cache: the keys and values saved for every token in the conversation so far, so they aren't recomputed. It grows by one token per step. GPU 7 · B200 Weights: the model's trained parameters. Every decode step reads all of them. KV cache: the keys and values saved for every token in the conversation so far, so they aren't recomputed. It grows by one token per step. NVLink reaches only the 8 GPUs in this node. Anything going to the other node leaves through a NIC and the InfiniBand rails. NVLink through NVSwitch, inside this node only NIC: the network card that connects one GPU to the InfiniBand fabric. NIC 0 NIC: the network card that connects one GPU to the InfiniBand fabric. NIC 1 NIC: the network card that connects one GPU to the InfiniBand fabric. NIC 2 NIC: the network card that connects one GPU to the InfiniBand fabric. NIC 3 NIC: the network card that connects one GPU to the InfiniBand fabric. NIC 4 NIC: the network card that connects one GPU to the InfiniBand fabric. NIC 5 NIC: the network card that connects one GPU to the InfiniBand fabric. NIC 6 NIC: the network card that connects one GPU to the InfiniBand fabric. NIC 7 Leaf switch: an InfiniBand switch for one rail. NIC i of every node plugs into leaf i, so GPU i in one node reaches GPU i in another through a single switch. rail 0 leaf Leaf switch: an InfiniBand switch for one rail. NIC i of every node plugs into leaf i, so GPU i in one node reaches GPU i in another through a single switch. rail 1 leaf Leaf switch: an InfiniBand switch for one rail. NIC i of every node plugs into leaf i, so GPU i in one node reaches GPU i in another through a single switch. rail 2 leaf Leaf switch: an InfiniBand switch for one rail. NIC i of every node plugs into leaf i, so GPU i in one node reaches GPU i in another through a single switch. rail 3 leaf Leaf switch: an InfiniBand switch for one rail. NIC i of every node plugs into leaf i, so GPU i in one node reaches GPU i in another through a single switch. rail 4 leaf Leaf switch: an InfiniBand switch for one rail. NIC i of every node plugs into leaf i, so GPU i in one node reaches GPU i in another through a single switch. rail 5 leaf Leaf switch: an InfiniBand switch for one rail. NIC i of every node plugs into leaf i, so GPU i in one node reaches GPU i in another through a single switch. rail 6 leaf Leaf switch: an InfiniBand switch for one rail. NIC i of every node plugs into leaf i, so GPU i in one node reaches GPU i in another through a single switch. rail 7 leaf InfiniBand: NIC i of every node plugs into leaf switch i, at most 50 GB/s each way between two nodes: NIC, leaf switch, NIC, 7 to 27 times longer (predicted) KV The request ('The capital of France is'), followed through the system. Blue on the way in, green once tokens head back.
Not sent yet
KV cache
Output

The request

1 of 18

What the request looks like

Before any GPU work, a request has a shape: how many tokens go in, how many come out, and how fast the user needs them. Chat, reasoning, summarization and agents have very different shapes, and each layer below matters more for some of them than for others.

126k read back, 857 appended tokens read back vs tokens added in a median Claude Code step From Inference in production (part 5): workload types
Go deeper:
2 of 18

Picking a replica

A gateway takes the request and a router picks which copy of the model (a replica) serves it. Before answering, a replica runs prefill, one pass over the whole prompt that saves each token's keys and values: the KV cache. A router that knows which replica already holds this prompt's KV cache can skip most of that work.

0.54 s vs 93 s p90 time to first token, precise cache-aware routing vs random, on llm-d's shared-prefix benchmark From Inference in production (part 2): the gateway, the router and the autoscaler
Go deeper:

One replica

3 of 18

The engine

Each replica runs an engine such as vLLM or SGLang. Its API server takes the request, the scheduler batches it with other requests, the KV cache manager finds room for its cache, and the model runner does the GPU work.

about 121 requests Qwen3-8B on one B200 has KV cache room for about 121 requests at 8k context (arithmetic) From Inference engines (part 2): deploying one
Go deeper:
4 of 18

Kernels

The GPU runs small programs called kernels, one after another: matrix multiplies, adds, and the handful of kernels that make up each norm. The CPU launches each one, and when the GPU finishes a kernel before the next launch arrives, it waits. This measurement used Hugging Face transformers, which launches every kernel from Python; an engine like vLLM runs the same model with fewer, larger kernels and replays them with CUDA graphs.

2,398 kernels to make one token with Qwen3-8B on an H100 (Hugging Face transformers, one user) From Kernels: what they are and why inference cares
Go deeper:
5 of 18

Many users at once

After prefill, the model makes the answer one decode step at a time, one new token per request per step. In a dense model like Qwen3-8B, each step reads every weight once, however many requests share it, so the engine puts many users' next tokens in the same step. The first users added cost almost nothing, until the compute catches up and every step gets slower.

$7.59 to $0.15 GPU cost per million output tokens on one H100, at 1 user and at 128 users, the largest tested batch under 20 ms per token (Qwen3-8B, vLLM) From Batching: how one GPU serves many users, and what a token costs
Go deeper:

Inside the GPU

6 of 18

Prefill, decode and the KV cache

Prefill runs the whole prompt through the model and stores each token's keys and values: the KV cache. Engines can split a long prompt into chunks, as stop 10 shows. Decode then produces one token per call, reusing the cache and adding one token's worth to it each time.

23,040 bytes of KV cache per token for SmolLM2-135M (arithmetic from its config) From Watching a KV cache grow (part 1): what's in it and how big it gets
Go deeper:
7 of 18

Why decode gets slower

Every decode step reads the model's weights and the whole KV cache out of memory. As the conversation grows, so does the cache, and each new token takes longer.

18 ms → 12.6 ms per token at 32k context for SmolLM2-135M on an M4 Pro with the default cache, before and after removing its extra cache copies (one run; 32k is past its 8k trained length) From Watching a KV cache grow (part 2): why decode gets slower
Go deeper:
8 of 18

Guessing ahead

With one user, a decode step leaves the GPU's compute mostly idle while it waits for the weights. Speculative decoding fills that time: something cheap guesses the next few tokens, and the model checks them all in one step, keeping every guess up to the first one it disagrees with.

1.66x more tokens per second for one user on GSM8K math questions, with Qwen3-0.6B guessing 4 tokens a step for Qwen3-8B, greedy decoding (H100, vLLM) From Speculative decoding: guessing tokens to use idle compute
Go deeper:
9 of 18

Weight precision

Every weight is stored with a fixed number of bits, from 16 in BF16 down to 4 in FP4. Fewer bits means fewer GPUs to hold the model, more room for the KV cache and, on GPUs that compute in that format, faster tokens, at some cost in accuracy.

140 GB to 35 GB a 70B model's weights in BF16 vs FP4, before FP4's scales add a few GB (arithmetic; Qwen3-8B's real checkpoint shrank 2.6x, not 4x) From Weight precision (part 1): how it affects serving
Go deeper:
10 of 18

Prefill stalls decode

When a long prompt is prefilled on the GPU where other users are decoding, their tokens stop until that prefill finishes. Chunking the prefill trades one long stall for many short ones, and running prefill on a separate GPU removes the stall, as long as handing over the cache doesn't block decode.

1.5 s the stall another user sees while a 32k-token prompt prefills with chunked prefill off; 512-token chunks cut the longest gap to 111 ms (H100, Qwen2.5-7B, a Python loop on Transformers, not a serving engine) From Inference disaggregation (part 1): why split prefill and decode
Go deeper:

Between GPUs

11 of 18

Moving the cache

Once prefill and decode run on different GPUs, the KV cache has to travel between them. Inside one machine over NVLink that takes a few milliseconds. Across a network it is predicted to take 7 to 27 times longer; that hasn't been measured yet.

5.7 ms to move a 1.8 GiB cache over NVLink, half a percent of the prefill that built it (2 H100s, Qwen2.5-7B) From Inference disaggregation (part 2): moving the cache
Go deeper:
13 of 18

Back to the user

Each token streams back through the router and gateway (in some stacks one program) to the chat as soon as it is made. Benchmarks split the wait in two: TTFT, the time to the first token, which is mostly queueing plus prefill, and TPOT, the average gap between the tokens after it, which is mostly decode plus any stalls from other requests' prefills.

0.23 s, then 6.8 ms TTFT and mean TPOT for 1,024 tokens in and 1,024 out, 16 at a time, chunked prefill and prefix caching off (Qwen2.5-7B, one H100) From Inference in production (part 5): workload types
Go deeper:
14 of 18

The next turn

A chat app sends the whole conversation again on every turn, and an agent resends its context on every call. If the server still has the KV cache for the part it has seen, it only prefills what's new. That's prefix caching. Only a matching start counts: the tokens must be the same from the first one, and the cache lasts until it is evicted to make room.

30 ms vs 0.5 s to the first token for a reused 16k-token document vs a new one (Qwen3-8B, H100) From Watching a KV cache grow (part 4): reusing it
Go deeper:

Running it

15 of 18

Adding replicas

When traffic grows, the autoscaler adds replicas. Without its compile caches, a new replica takes minutes before it serves anything, most of it spent compiling kernels. Keeping the caches cut a 7B's start to about a minute and a 32B's to about two. Either way scaling has to start before the queue is already long.

200 s for a new 7B replica without compile caches to pass its health check (68 s with them kept; image and weights already on the node) From Inference in production (part 2): the gateway, the router and the autoscaler
Go deeper:
16 of 18

When hardware fails

At scale, hardware fails often. How many GPUs one failure takes down depends on the NVLink domain and the replica size. Requests running on those GPUs lose their KV cache, so another replica has to prefill them again before it can carry on.

every 3 hours one interruption, from any cause, on 16,384 H100s during Llama 3 training From Inference in production (part 3): when hardware goes bad
Go deeper:
17 of 18

Rolling out a new model

A rolling update restarts every replica on purpose. Draining requests in flight, canaries, and keeping prefill and decode on the same model version keep users from noticing.

75% fleet capacity while eight 32B replicas roll two at a time, with no spare GPUs to surge onto (arithmetic) From Inference in production (part 4): rolling out a new model
Go deeper:

Planning

18 of 18

How many GPUs?

Sizing uses every layer above. Traffic and latency targets give the requests in flight, memory and decode speed give requests per replica, prefill takes its share, and then you add spares.

6 H100s or 4 B200s for Qwen3-8B at 20 requests a second, 2,000 tokens in and 400 out, at most 20 ms per token and the first token within a second, including 2 spares (arithmetic, not a load test) From Sizing inference: from traffic to a GPU count
Go deeper: