4. Positional encoding and RoPE

Raw self-attention is permutation-invariant: shuffle the tokens and the maths is identical. Position information must be added explicitly.

Approaches, in historical order: - Sinusoidal absolute encodings (original transformer) — fixed vectors added to embeddings. - Learned absolute positions — an embedding table per index; poor extrapolation beyond training length. - RoPE (Rotary Position Embedding) — rotates query/key vectors by an angle proportional to position, so attention depends on *relative* distance. Dominant in modern open models. - ALiBi — adds a distance-proportional bias to attention scores.

Context extension usually means rescaling RoPE frequencies (NTK-aware scaling, YaRN) plus a short fine-tune on long documents.

python
# RoPE in spirit: rotate pairs of dimensions by position * theta
def rope(x, pos, theta):
    even, odd = x[..., 0::2], x[..., 1::2]
    c, s = cos(pos * theta), sin(pos * theta)
    return concat([even * c - odd * s, even * s + odd * c], dim=-1)

Gotcha: a model advertised at 1M tokens may still show degraded recall in the middle of the window — always evaluate on your own long-context task.

Read in another language

How to study this topic

Scoring

Check your understanding

Badges