https://hiren.me/tags/speculative-decoding/ Recent content in Speculative Decoding on Hugo en Thu, 08 Oct 2026 17:10:00 -0700 https://hiren.me/posts/speculative-decoding/ Thu, 08 Oct 2026 17:10:00 -0700 https://hiren.me/posts/speculative-decoding/ <p>With one user, a GPU spends most of each decode step waiting for the weights to arrive from memory, and its math units sit mostly idle. <a href="https://hiren.me/posts/batching/" >Batching</a> fills that idle math with other users&rsquo; tokens. Speculative decoding fills it with guesses: something cheap guesses the next few tokens, and the model checks all of them in one step. When the guesses are right, one step produces several tokens. This post measures how often the guesses are right, and what that does to speed on an H100.</p>