A reference stack
Edge: DNS with health checks, CDN and WAF. Ingress: layer 7 load balancer terminating TLS. Compute: containers on Kubernetes or a managed runtime, spread across three AZs. State: managed relational primary with read replicas, plus object storage and a cache. Async: queue and workers. Everything defined in IaC and deployed by pipeline.
Failure modes to design for
AZ loss, dependency latency spikes, thundering-herd retries, cache stampedes, database connection exhaustion, expired certificates and quota limits. Assume every remote call can hang; enforce timeouts, retry budgets with jitter, and circuit breakers.
Production readiness
Before launch: SLOs and dashboards, alert routing and on-call rota, tested backups and a rehearsed restore, autoscaling verified under load test, least-privilege IAM reviewed, secrets rotating, cost budget alarms, and a documented rollback.
- Backups are only real once a restore has been rehearsed.
- Load test before launch, not after the incident.
- Every dependency needs a timeout.