Medallion layers
Bronze holds raw ingested data exactly as received, so any bug is replayable. Silver is cleaned, typed, deduplicated and conformed. Gold contains business-level aggregates and marts that analysts and dashboards consume. Each layer has a clear owner and contract.
Table formats and catalogues
Parquet is the columnar file format; Delta, Iceberg and Hudi add a transaction log giving ACID commits, schema evolution, partition evolution and time travel. A catalogue (Glue, Unity, Hive Metastore) tracks tables, lineage and permissions.
Quality is a pipeline stage
Test data like code: not-null and uniqueness on keys, referential integrity, accepted value sets, freshness and row-count anomaly checks. Publish data contracts between producers and consumers so a schema change upstream is a breaking change, not a silent one.
- Fail loudly and quarantine bad batches.
- Track lineage so impact is knowable.
- Freshness SLAs matter as much as correctness.