Optimisers and schedules
SGD with momentum still generalises beautifully; AdamW is the pragmatic default because it adapts per-parameter step sizes and decouples weight decay. The learning rate is the highest-leverage hyperparameter — warm it up, then decay it (cosine or step).
Normalisation and residuals
Batch normalisation stabilises convolutional nets; layer normalisation does the same for transformers where batch statistics are unreliable. Residual connections let gradients skip layers, which is what made very deep networks trainable at all.
Regularisation and precision
Dropout, weight decay, data augmentation and early stopping all limit memorisation. Mixed-precision (bf16/fp16) training roughly halves memory and greatly increases throughput on modern accelerators, with gradient scaling to avoid underflow.
- Loss spikes → lower LR or add warmup.
- Train loss falls, val rises → regularise.
- NaNs in fp16 → use bf16 or a grad scaler.