Batching: how one GPU serves many users, and what a token costs

Thu, 08 Oct 2026 17:01:00 -0700

A hosted model answers thousands of people at once, and a single user still gets their tokens quickly. Both come from batching, where one GPU works on many requests in the same step. This post explains why batching is nearly free at first, measures what happens as one H100 takes on more users, and turns the result into the cost of a token.

Why batching is nearly free at first Link to heading

Generating a token reads the model’s weights from GPU memory. Kernels worked out what that costs for Qwen3-8B on an H100: each step reads 15.1 GB of weights, and the memory moves 3.35 TB/s, so a step takes at least 4.5 ms. The math in that step takes a small fraction of the time; the GPU mostly waits for the weights to arrive.

Cost on

Batching: how one GPU serves many users, and what a token costs

Why batching is nearly free at first Link to heading