https://hiren.me/tags/kernels/ Recent content in Kernels on Hugo en Wed, 07 Oct 2026 17:52:48 -0700 https://hiren.me/posts/kernels/ Wed, 07 Oct 2026 17:52:48 -0700 https://hiren.me/posts/kernels/ <p>An LLM running on a GPU spends its time in kernels, the small programs the GPU runs one after another. In the measurement later in this post, generating one token with Qwen3-8B on an H100 took 2,398 of them. This post starts from the beginning: what a kernel is, how model code turns into kernels, where kernels come from, and which ones run for each token.</p> <h2 id="why-a-gpu"> Why a GPU <a class="heading-link" href="#why-a-gpu"> <i class="fa-solid fa-link" aria-hidden="true" title="Link to heading"></i> <span class="sr-only">Link to heading</span> </a> </h2> <p>Generating one token means multiplying a vector by every weight matrix in the model. Each output number is a long sum of multiplications, and every output number can be worked out at the same time as the others. A CPU has a handful of powerful cores and does a few of these at a time. A GPU has thousands of simpler ones and does many at once.</p>