Designing an A/B test
Fix the hypothesis, the primary metric and the minimum detectable effect before launch, then compute the sample size and run for a whole number of business cycles. Randomise at the right unit — usually the user, not the session — and validate with an A/A test.
Traps
Peeking at results and stopping when significance appears massively inflates false positives; use fixed horizons or sequential tests designed for it. Watch for sample ratio mismatch, novelty effects, and network interference where treated users influence controls.
When you cannot randomise
Difference-in-differences compares treated and untreated groups before and after; regression discontinuity exploits a threshold rule; instrumental variables and synthetic controls handle harder cases. All rest on assumptions you must state and defend.