Inference engines (part 3): the stacks you can't see

Mon, 05 Oct 2026 16:20:00 -0700

Part 1 covered what an engine is, and part 2 how to deploy one. The alternative is calling an API, where the provider picks the engine, the GPUs and the number formats. This post maps those options, what each provider says about its stack, and how to evaluate one you can’t inspect.

Ways to serve a model. From left to right, you run less of the stack yourself, and you can check less of what runs it.

What providers say about their stacks Link to heading

Provider	What they say	Source
Baseten	“we benchmark frameworks like TensorRT-LLM, SGLang, and vLLM to select the best-performing framework”	guide, Jan 2026
Anyscale	“Ray Serve for orchestration and scaling. vLLM for inference.”	docs
DeepInfra	“our inference stack is built on TensorRT-LLM and NVIDIA Dynamo”	blog, Jun 2026
Fireworks	“Fireworks proprietary LLM serving stack, which consists of CUDA kernels, optimized for both FP16 and FP8”	blog, Jan 2024
Together	the Together Inference Engine builds on “FlashAttention-3, faster GEMM & MHA kernels, innovations in quality-preserving quantization, and speculative decoding”	blog, Jul 2024
Groq	“Data flow is statically scheduled by the software during compilation”	blog, Mar 2025
Cerebras	“we are able to integrate 44GB of SRAM on a single chip”	blog, Aug 2024

Fireworks’ current inference page says “We built it from the ground up, optimized every layer we control, from GPU memory layout to the runtime”. Whether any of it started from an open engine, neither Fireworks nor Together says.

Together on

Inference engines (part 3): the stacks you can't see

What providers say about their stacks Link to heading