Reusing the KV cache
The invariant that makes reuse legal
Decode's attention was unfolded in §4 of the first post: the causal mask kills every column past $n$, the softmax is row-wise, and row $n$ of the output collapses to a matrix–vector product. At step $n$, one head of one layer computes
\begin{equation} \vect{o} \;=\; \softmax\!\left(\frac{\vect{q}\,\mat{K}_{1:n}\T}{\sqrt{\dhead}}\right)\mat{V}_{1:n}, \qquad \vect{q} \defeq \mat{Q}_{n,:} \in \R^{1\times\dhead} \end{equation}Only $\vect{q}$ and the new rows $\vect{k}_n$, $\vect{v}_n$ are computed at this step; $\mat{K}_{1:n}$ and $\mat{V}_{1:n}$ are read in full. That is what the cache holds.
And they are only ever appended to. Row $i$ of the layer-$\ell$ cache is
\begin{equation} \vect{k}_i = \Rope_i\!\big(\vect{x}^{(\ell-1)}_i\mat{W}_K\big), \qquad \vect{v}_i = \vect{x}^{(\ell-1)}_i\mat{W}_V \end{equation}and $\vect{x}^{(\ell-1)}_i$ is a function of $t_1,\dots,t_i$ alone: causal attention lets no later position reach it, and every other stage in the block is row-wise. So step $n$ writes one new row and leaves the others untouched,
\begin{equation} \mat{K}_{1:n} = \begin{bmatrix} \mat{K}_{1:n-1} \\ \vect{k}_n \end{bmatrix}, \qquad \mat{V}_{1:n} = \begin{bmatrix} \mat{V}_{1:n-1} \\ \vect{v}_n \end{bmatrix} \end{equation}Whether row $i$ was written during prefill or during an earlier decode step, it is the same row, and it is the row every later step reads. Nothing recomputes it to a different value. Stated for the cache as a whole, with $\KVc_{1:S}$ the first $S$ entries over all layers and heads:
\begin{equation} \KVc_{1:S}\big(t_{1:T}\big) \;=\; \KVc_{1:S}\big(t_{1:S}\big) \qquad\text{for every } S \le T \end{equation}The cache is prefix-closed: what follows a prefix never changes the prefix's entries. So they stay valid for the rest of this sequence, and for any other sequence starting the same way.