Tokens and positions
Text is split by a subword tokeniser (BPE or SentencePiece) into token ids, embedded into vectors, and given positional information — learned embeddings originally, rotary embeddings (RoPE) in most current models.
Self-attention
Each token projects to a query, key and value. Scores are query·key scaled by the square root of the head dimension, softmaxed, then used to blend values. Multiple heads run in parallel so different heads can track syntax, coreference or position.
import torch, math
def attention(q, k, v, mask=None):
scores = q @ k.transpose(-2, -1) / math.sqrt(q.size(-1))
if mask is not None:
scores = scores.masked_fill(mask == 0, float('-inf'))
return torch.softmax(scores, dim=-1) @ vThe block and the stack
A block is attention plus a feed-forward MLP, each wrapped in residual connections and layer norm. Decoder-only models mask future positions so a token can only attend backwards, which makes next-token prediction well defined. Context length is limited because attention cost grows quadratically with sequence length.