DAGs and schedulers
Orchestrators (Airflow, Dagster, Prefect) model work as a directed acyclic graph of tasks with dependencies, schedules and retries. Tasks should be small, idempotent and parameterised by the logical execution date rather than 'now'.
Failure handling
Configure retries with backoff for transient failures, timeouts so a hung task cannot block the schedule, and alerts on SLA misses. Sensors that wait for upstream data should time out rather than occupy a worker indefinitely.
Observability of data
Track per-run row counts, freshness, schema changes and runtime. A pipeline that succeeds while producing zero rows is a failure that green dashboards will happily hide.