← Blog

How GPUs work, with the B200

2026.08.20 · Hyeondo Jang

Contents

  1. 1The memory hierarchy
  2. 2Compute-bound and memory-bound
  3. 3The roofline

1The memory hierarchy

A B200 is 160 copies of one unit, the streaming multiprocessor. Figure 1 draws one of them and everything underneath it.

Worth settling first what a GPU does not worry about. Each of an SM's four subpartitions holds many warps — groups of 32 threads issued together — and its scheduler switches to another warp on any cycle the current one is waiting, so a slow access stalls nothing. Latency is covered by having other work outstanding. Bandwidth cannot be covered that way: if the bytes are not arriving fast enough, more warps ready to run does not help, because they are all waiting on the same pipe. That asymmetry is why the rest of this post looks at one axis only.

One SM — × 160 on the chip subpartition 0 warp scheduler 32 FP32 cores 1 tensor core 64 KB reg — 100+ TB/s subpartition 1 warp scheduler 32 FP32 cores 1 tensor core 64 KB reg — 100+ TB/s subpartition 2 warp scheduler 32 FP32 cores 1 tensor core 64 KB reg — 100+ TB/s subpartition 3 warp scheduler 32 FP32 cores 1 tensor core 64 KB reg — 100+ TB/s tens of TB/s L1 + shared memory — 256 KB, SRAM one block per SM, split between the two per kernel · 41 MB across the chip ≈ 12 TB/s L2 — 126 MB, SRAM outside every SM, shared by all 160, still on the compute die 8 TB/s, through the HBM controllers HBM3E — 192 GB, DRAM stacked dies beside the compute die · this is where the model lives On-chip SRAM is 208 MB in total. Qwen3-8B is 8.2 GB in FP8, so it only fits in the bottom row.
Figure 1. The B200 memory hierarchy, drawn to make one point: every level fast enough to feed the tensor cores is far too small to hold a model, and the only level large enough is the slowest one.

Four numbers describe the whole hierarchy, and they are the only four worth memorising. Registers and L1 are per SM, so their chip-wide totals come from multiplying by 160:

Level Where it sits Per SM Whole chip Bandwidth
Registers inside each subpartition 256 KB 41 MB 100+ TB/s
L1 + shared one block per SM 256 KB 41 MB tens of TB/s
L2 on the compute die, shared — 126 MB ≈ 12 TB/s
HBM3E stacked beside the die — 192 GB 8 TB/s

Read the last two columns together and the figure's shape falls out. Capacity climbs by a factor of 1500 in a single step at the bottom — 126 MB of L2, then 192 GB of HBM — while bandwidth falls the whole way down. There is no level that is both fast and large, and the jump happens in the wrong place: exactly where a model has to live.

How badly it lands is worth stating in units of a real model. Qwen3-8B quantized to FP8 is 8.2 GB — one byte per parameter — which is about 40× all on-chip SRAM combined. L2 holds 1.5% of it. And it is not as though a well-written kernel could stage the model layer by layer: one Qwen3-8B decoder block is 193M parameters, or 193 MB in FP8 — still more than L2 can hold. Not one layer fits, let alone 36.

So there is no choice to make about where the weights live. They live in HBM, and every forward pass has to pull them up through the bottom arrow of the figure.

2Compute-bound and memory-bound

Weights arrive at the tensor cores at 8 TB/s whatever you are running. What differs is how much arithmetic each arriving byte brings with it — and that alone decides which side is idle.

same pipe, same rate — 8 TB/s tensor core, over time Prefill — one weight tile serves T tokens HBM SM tensor core compute-bound — busy 100%, the pipe is what waits Decode — one weight tile serves 1 token HBM SM tensor core memory-bound — busy 0.4%, the core is what waits Both lanes are fed at the same 8 TB/s. Only the work each delivered byte carries differs — 562 FLOP per byte is where the two exactly balance.
Figure 2. The same pipe, two workloads. Prefill keeps the core busy between arrivals; decode empties it. The tick strip is drawn at 14% for legibility — the real decode duty cycle is 0.4%.

Prefill pushes $\Lctx$ tokens through the same weight tile, so a tile bought once is multiplied $\Lctx$ times over. The core is still working when the next tile lands, so the pipe is what waits: compute-bound.

Decode pushes one token through it. The tile arrives, is used once, and the core is finished long before the next tile shows up. Now the core waits: memory-bound. Adding tensor cores would change nothing — they are not what is missing.

562 is the crossing point between the two lanes. It is how much arithmetic one byte has to carry for the core's work to last exactly as long as the wait for the next byte:

\begin{equation} \ridge \;=\; \frac{\peak}{\bw} \;=\; \frac{4500\ \mathrm{TFLOP/s}}{8\ \mathrm{TB/s}} \;=\; \mathbf{562\ \FLOP/\Bytes} \end{equation}

Prefill over a 40k prompt brings tens of thousands of FLOPs per byte. Decode in FP8 brings two — one multiply and one add per weight byte, because a weight is one byte and it is used once. That factor is the entire distance between the two lanes above.

3The roofline

The same statement, plotted. A kernel's arithmetic intensity $\AI$ is the FLOPs it performs per byte it moves; neither budget can be exceeded, so the attainable rate is whichever runs out first:

\begin{equation} \text{attainable FLOP/s} \;=\; \min\big(\peak,\; \bw \cdot \AI\big) \label{eq:roofline} \end{equation}

Below $\ridge$ the second term binds and performance is a straight line in $\AI$; above it the first binds and extra intensity buys nothing. $\bw$ is HBM's 8 TB/s: a byte crosses three links to reach a core, but they run in series, so the narrowest one sets the rate.

Batch-1 decode reads 8.2 GB of weights at two FLOPs per parameter, so $\AI = 2$ — more than 280× to the left of the ridge. By \eqref{eq:roofline} it attains $8 \times 2 = 16$ TFLOP/s, 0.36% of peak, and no amount of kernel engineering moves it, because the limit is not in the kernel.

1 10 10² 10³ 10⁴ 10⁵ 1 10 100 1000 10⁴ ridge I* = 562 decode, batch 1 — I = 2, 16 TF/s prefill 280× short B200 (4500 TF/s, 8 TB/s) H100 (1979 TF/s, 3.35 TB/s) RTX 5090 (419 TF/s, 1.79 TB/s) arithmetic intensity I (FLOP/byte) attainable TFLOP/s
Figure 3. Rooflines for three GPUs, all quoted as dense FP8. The sloped section is the bandwidth wall, the flat section the compute ceiling, and the corner is the ridge point — 562 for the B200, 591 for the H100, 234 for the RTX 5090. Prefill runs on the ceiling; batch-1 decode runs near the far left of every slope.