https://hiren.me/tags/fp8/ Recent content in FP8 on Hugo en Sun, 04 Oct 2026 11:00:00 -0700 https://hiren.me/posts/weight-precision-part-1/ Sun, 04 Oct 2026 11:00:00 -0700 https://hiren.me/posts/weight-precision-part-1/ <p>FP32, BF16, FP8 and FP4 turn up all over LLM serving. A checkpoint is called <code>Qwen3-8B-FP8</code> or <code>Qwen3-8B-NVFP4</code>. vLLM takes flags like <code>--dtype bfloat16</code> and <code>--kv-cache-dtype fp8</code>. A GPU datasheet lists a different speed for BF16, FP8 and FP4. Each is a number format: it sets how many bits store every number in the model, from 32 in FP32 down to 4 in FP4. The choice decides how many GPUs a model needs, how many users fit, how fast tokens come out, and how accurate the answers are.</p>