3. The transformer and self-attention

The transformer replaced recurrence with self-attention. Each token is projected into a query, key, and value vector; the attention score between two tokens is the scaled dot product of query and key, softmaxed into weights over the values.

Key building blocks of a decoder block: - Multi-head attention — several attention heads learn different relations (syntax, coreference, position). - Causal mask — a token may only attend to earlier positions, which is what makes generation possible. - Feed-forward network (MLP) — per-token nonlinear transformation, usually 4x the hidden width. - Residual connections + normalisation (RMSNorm) — keep very deep stacks trainable.

Attention is O(n²) in sequence length, which is why long contexts are expensive and why FlashAttention, sliding-window attention, and KV-cache tricks exist.

Gotcha: "bigger context" is not free — doubling the prompt roughly quadruples attention work in the prefill phase.