From flat files to relations
Early systems stored records in application-specific files, so every program re-implemented access logic. Codd's 1970 relational model separated the logical view of data from physical storage, and SQL made that view queryable declaratively. That separation is why relational databases outlived their competitors.
Warehouses, lakes, lakehouses
The 1990s added the data warehouse: modelled, cleaned, analytics-oriented storage separate from operational systems. Hadoop-era data lakes stored raw files cheaply but degraded into swamps without schema or governance. The lakehouse combines lake storage with table formats (Delta, Iceberg, Hudi) that add ACID transactions, schema evolution and time travel.
Who does what
Data engineers build and operate pipelines and models; analytics engineers turn raw tables into trusted, documented marts; analysts answer business questions; data scientists build statistical and ML models; ML engineers productionise them. The boundaries blur, the ordering does not — nothing downstream works without reliable pipelines.
- OLTP: many small transactions, row-oriented.
- OLAP: few huge scans, column-oriented.
- Lakehouse: lake storage plus warehouse guarantees.