How LLMs work, with Qwen3-8B
Contents
1Prefill and decode: what runs when you send a prompt
Inference runs in two phases. A question comes in, and processing it is prefill. Generating the answer’s tokens, one at a time, is decode.
2The Qwen3-8B architecture
One forward pass, end to end
CLOSE ✕The stack is fixed-width. Every stage inside a decoder block maps a $\Lctx \times \dmodel$ matrix to another matrix of the same shape, with $\dmodel = 4096$ for Qwen3-8B. Only the two ends of the figure change shape: the tokenizer produces integers, and the LM head widens the last axis to the vocabulary.
3Deep dive: one prefill pass
The stages below are the operators of Figure 2, written out one at a time for a prefill pass — all $\Lctx$ prompt tokens entering at once. §4 then says what decode does differently.
Tokenizer
The tokenizer cuts the prompt into $\Lctx$ sub-word pieces drawn from a fixed vocabulary of size $\Vocab = 151{,}936$, so what the model actually receives is a list of $\Lctx$ integers $(t_1,\dots,t_{\Lctx})$ — no floating-point arithmetic has happened yet.
Embedding
Each id indexes one row of an embedding table $\mat{E}\in\R^{\Vocab\times\dmodel}$. The embedding layer is a row lookup — no arithmetic at all:
\begin{equation} \mat{X}^{(0)} \;=\; \begin{bmatrix} \mat{E}_{t_1} \\ \vdots \\ \mat{E}_{t_{\Lctx}} \end{bmatrix} \;\in\; \R^{\Lctx \times \dmodel} \end{equation}One row per token, one column per feature. This is the shape everything downstream preserves, and it is called the residual stream.
RMSNorm
Each row is rescaled to a fixed norm before it enters attention or the feed-forward network. With a learned gain $\vect{g}\in\R^{\dmodel}$ and elementwise product $\odot$:
\begin{equation} \RMSNorm(\vect{x}) \;=\; \frac{\vect{x}}{\sqrt{\dfrac{1}{\dmodel}\displaystyle\sum_{i=1}^{\dmodel} x_i^{2}}} \;\odot\; \vect{g} \end{equation}It acts on each row independently and moves no tokens, so it costs almost nothing.
Multi-head attention (grouped-query attention)
Original multi-head attention — K and V are 4096 wide, one pair per head
CLOSE ✕This is the only stage where information moves between token positions. Each of the $\nq = 32$ heads projects the same input into a query, a key and a value of width $\dhead = 128$, then mixes rows with a causal mask $\mat{M}$ that sets $M_{ij} = -\infty$ for $j > i$:
\begin{equation} \mat{H}_i \;=\; \Attn\!\left(\mat{X}\mat{W}_Q^{(i)},\; \mat{X}\mat{W}_K^{(i)},\; \mat{X}\mat{W}_V^{(i)}\right), \qquad i = 1,\dots,\nq \end{equation} \begin{equation} \Attn(\mat{Q},\mat{K},\mat{V}) \;=\; \softmax\!\left(\frac{\mat{Q}\mat{K}\T}{\sqrt{\dhead}} + \mat{M}\right)\mat{V} \end{equation}The heads never see each other inside the attention; the output projection $\mat{W}_O$ is what combines what they found:
Qwen3-8B does not give every head its own keys and values: it has $\nq = 32$ query heads but only $\nkv = 8$ key/value heads, so each $\mat{K},\mat{V}$ pair is shared by four query heads. That variant is grouped-query attention (GQA) — the same operator as above with $\mat{K}_{g(i)},\mat{V}_{g(i)}$ in place of $\mat{K}_i,\mat{V}_i$, where $g(i) = \lceil i / (\nq/\nkv) \rceil$ is the group head $i$ belongs to. Full multi-head attention is the special case $\nkv = \nq$.
\begin{equation} \MHA(\mat{X}) \;=\; \big[\,\mat{H}_1 \;\|\; \cdots \;\|\; \mat{H}_{\nq}\,\big]\,\mat{W}_O, \qquad \mat{W}_O \in \R^{\nq \dhead \times \dmodel} \end{equation}Feed-forward network
Qwen3 uses SwiGLU, three matrices per block, with inner width $\dff = 12{,}288 = 3\dmodel$:
\begin{equation} \FFN(\mat{X}) \;=\; \big[\, \silu(\mat{X}\mat{W}_{\mathrm{gate}}) \odot \mat{X}\mat{W}_{\mathrm{up}} \,\big]\, \mat{W}_{\mathrm{down}} \end{equation}Unlike attention, the feed-forward network acts on each row on its own — token $i$'s output depends only on token $i$'s input. It is also where most of the parameters live.
The block, and the head
Block $\ell$ adds two corrections to the residual stream rather than replacing it:
\begin{equation} \mat{X}' \;=\; \mat{X}^{(\ell-1)} + \MHA\!\left(\RMSNorm(\mat{X}^{(\ell-1)})\right) \end{equation} \begin{equation} \mat{X}^{(\ell)} \;=\; \mat{X}' + \FFN\!\left(\RMSNorm(\mat{X}')\right) \end{equation}After $\nlayer = 36$ such blocks, the LM head normalizes once more and projects onto the vocabulary, giving one score per vocabulary entry per position:
\begin{equation} \text{logits} \;=\; \RMSNorm\!\left(\mat{X}^{(\nlayer)}\right) \mat{W}_{\mathrm{lm}} \;\in\; \R^{\Lctx \times \Vocab} \end{equation}4Decode: what changes
§3 is written for prefill, where all $\Lctx$ prompt tokens go through the stack at once. Decode runs the same weights through the same code on a single row, and most of the figure simply narrows: the tokenizer emits one id, the embedding returns one row, RMSNorm rescales that row, and the feed-forward network — which acts on each row independently — sees a $1\times\dmodel$ input. None of those operators can tell the difference.
Attention is the exception, because it is the only operator that mixes rows. Write $\mat{O} = \softmax\!\big(\mat{Q}\mat{K}\T/\sqrt{\dhead} + \mat{M}\big)\mat{V}$ for the prefill output of one head. At decode step $n$ the model needs only row $n$ of that matrix, and unfolding that row is what turns the operation into a matrix–vector product:
\begin{equation} \begin{aligned} \vect{o} \;=\; \mat{O}_{n,:} &\;=\; \Big[\softmax\!\Big(\tfrac{\mat{Q}\mat{K}\T}{\sqrt{\dhead}} + \mat{M}\Big)\mat{V}\Big]_{n,:} \\[3pt] &\;=\; \softmax\!\Big(\tfrac{(\mat{Q}\mat{K}\T)_{n,:}}{\sqrt{\dhead}} + \mat{M}_{n,:}\Big)\mat{V} \\[3pt] &\;=\; \softmax\!\Big(\tfrac{\mat{Q}_{n,:}\mat{K}\T}{\sqrt{\dhead}} + \mat{M}_{n,:}\Big)\mat{V} \\[3pt] &\;=\; \softmax\!\Big(\tfrac{\mat{Q}_{n,:}\mat{K}_{1:n}\T}{\sqrt{\dhead}}\Big)\mat{V}_{1:n} \\[3pt] &\;=\; \softmax\!\Big(\tfrac{\vect{q}\,\mat{K}_{1:n}\T}{\sqrt{\dhead}}\Big)\mat{V}_{1:n} \end{aligned} \end{equation}Line by line: the softmax is row-wise, so row $n$ of the output depends only on row $n$ of the scores; row $n$ of $\mat{Q}\mat{K}\T$ is $\mat{Q}_{n,:}\mat{K}\T$; the causal mask kills every column $j > n$, which drops $\mat{K}$ and $\mat{V}$ to their first $n$ rows and removes the mask altogether; and writing $\vect{q} \defeq \mat{Q}_{n,:} \in \R^{1\times\dhead}$ leaves
\begin{equation} \vect{q}\,\mat{K}_{1:n}\T \;\in\; \R^{1\times n} \qquad\text{where prefill had}\qquad \mat{Q}\mat{K}\T \;\in\; \R^{\Lctx\times\Lctx} \end{equation}so the score matrix has collapsed to a single row, and $\vect{o}\in\R^{1\times\dhead}$. Of the operands, only $\vect{q}$ and the new row of $\mat{K}$ and $\mat{V}$ are computed at this step — rows $1..n-1$ were produced on earlier steps and are read back rather than recomputed.
One more asymmetry, in the LM head rather than in an operator: prefill produces logits at every position and uses only the last row, while decode produces the one row it needs.