Weight precision (part 2): how FP4 works, and what it costs

Sun, 04 Oct 2026 12:00:00 -0700

Part 1 went down the ladder from FP32 to FP4 and what each step changes in production. This part stays on the last step. A weight stored in BF16 or FP16 can take any of 65,536 values. A weight stored in FP4 can take one of 16. Round Qwen3-8B’s weights to FP4 the obvious way, and all but 523 of the 50 million weights in one of its matrices become 0. Yet in one run, an FP4 version of the same model answered 1,205 of 1,319 grade-school math problems correctly, against 1,231 for the BF16 model.

Weight precision (part 1): how it affects serving

Sun, 04 Oct 2026 11:00:00 -0700

FP32, BF16, FP8 and FP4 turn up all over LLM serving. A checkpoint is called Qwen3-8B-FP8 or Qwen3-8B-NVFP4. vLLM takes flags like --dtype bfloat16 and --kv-cache-dtype fp8. A GPU datasheet lists a different speed for BF16, FP8 and FP4. Each is a number format: it sets how many bits store every number in the model, from 32 in FP32 down to 4 in FP4. The choice decides how many GPUs a model needs, how many users fit, how fast tokens come out, and how accurate the answers are.

FP4 on

Weight precision (part 2): how FP4 works, and what it costs

Weight precision (part 1): how it affects serving