Inference engines (part 3): the stacks you can't see

Mon, 05 Oct 2026 16:20:00 -0700

Part 1 covered what an engine is, and part 2 how to deploy one. The alternative is calling an API, where the provider picks the engine, the GPUs and the number formats. This post maps those options, what each provider says about its stack, and how to evaluate one you can’t inspect.

Ways to serve a model. From left to right, you run less of the stack yourself, and you can check less of what runs it.

What providers say about their stacks Link to heading

Provider	What they say	Source
Baseten	“we benchmark frameworks like TensorRT-LLM, SGLang, and vLLM to select the best-performing framework”	guide, Jan 2026
Anyscale	“Ray Serve for orchestration and scaling. vLLM for inference.”	docs
DeepInfra	“our inference stack is built on TensorRT-LLM and NVIDIA Dynamo”	blog, Jun 2026
Fireworks	“Fireworks proprietary LLM serving stack, which consists of CUDA kernels, optimized for both FP16 and FP8”	blog, Jan 2024
Together	the Together Inference Engine builds on “FlashAttention-3, faster GEMM & MHA kernels, innovations in quality-preserving quantization, and speculative decoding”	blog, Jul 2024
Groq	“Data flow is statically scheduled by the software during compilation”	blog, Mar 2025
Cerebras	“we are able to integrate 44GB of SRAM on a single chip”	blog, Aug 2024

Fireworks’ current inference page says “We built it from the ground up, optimized every layer we control, from GPU memory layout to the runtime”. Whether any of it started from an open engine, neither Fireworks nor Together says.

Inference engines (part 1): what an engine does and where it sits

Mon, 05 Oct 2026 16:00:00 -0700

vLLM, SGLang, TensorRT-LLM, Dynamo, llm-d, Fireworks. These names get compared as if they were the same kind of thing, but they sit at different layers. This post maps the layers, opens up the one layer they’re named after, the engine, and ends with what to check before running one in production. Part 2 covers deploying one, and part 3 the proprietary stacks you can’t see.

The layers Link to heading

The layers of a serving stack. Run it yourself and you pick each layer. Use a hosted API and the provider picks all of them.

A replica is one running copy of a model, on as many GPUs as that copy needs, which mostly comes down to memory. The engine runs one replica: it takes requests, batches them, and runs them on the GPUs. A service runs several replicas behind a router when it needs more traffic than one copy can serve, or has to survive a failure. Part 2 shows how to pick the number.

Serving on

Inference engines (part 3): the stacks you can't see

What providers say about their stacks Link to heading

Inference engines (part 1): what an engine does and where it sits

The layers Link to heading