1 · Transformers & Attention

1. Inside Self-Attention & RoPE Mechanics

The query-key-value dot product, causal masks, and Rotary Position Embeddings.

10 min read · 2 MCQs

The Self-Attention Equation

Self-attention allows every token in a sequence to dynamically attend to every other token. It computes $Attention(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$. The $\sqrt{d_k}$ scaling factor prevents dot products from growing excessively large in high dimensions, which would push softmax gradients into vanishing regions.

Rotary Position Embeddings (RoPE)

Unlike absolute positional embeddings added to input vectors, RoPE rotates Query and Key representations in the complex plane based on token positions. This preserves relative distance properties and enables context window extrapolation.

Chapter quiz

2 questions · pass mark 75%

1. Why are attention dot products scaled by the square root of the head dimension (sqrt(d_k))?
2. What is the primary advantage of Rotary Position Embeddings (RoPE)?