Inference engines (part 2): deploying one

Mon, 05 Oct 2026 16:10:00 -0700

Part 1 opened up an engine. This part runs one in production: what the pod looks like, what happens when the model doesn’t fit one GPU, and what to watch once it runs. The examples use vLLM on Kubernetes, with SGLang’s differences in a table. Part 3 covers stacks you can’t deploy yourself.

One replica Link to heading

One replica: a pod with one engine container on a GPU node. The model cache keeps weights across restarts. /dev/shm is shared memory the engine needs for tensor parallelism.

An engine ships as a container image: vllm/vllm-openai for vLLM, whose entrypoint is vllm serve. One replica is one pod running it. This is vLLM’s own Kubernetes example, cut down and pointed at Qwen3-8B:

Inference engines (part 1): what an engine does and where it sits

Mon, 05 Oct 2026 16:00:00 -0700

vLLM, SGLang, TensorRT-LLM, Dynamo, llm-d, Fireworks. These names get compared as if they were the same kind of thing, but they sit at different layers. This post maps the layers, opens up the one layer they’re named after, the engine, and ends with what to check before running one in production. Part 2 covers deploying one, and part 3 the proprietary stacks you can’t see.

The layers Link to heading

The layers of a serving stack. Run it yourself and you pick each layer. Use a hosted API and the provider picks all of them.

A replica is one running copy of a model, on as many GPUs as that copy needs, which mostly comes down to memory. The engine runs one replica: it takes requests, batches them, and runs them on the GPUs. A service runs several replicas behind a router when it needs more traffic than one copy can serve, or has to survive a failure. Part 2 shows how to pick the number.

SGLang on

Inference engines (part 2): deploying one

One replica Link to heading

Inference engines (part 1): what an engine does and where it sits

The layers Link to heading