The lineage
MapReduce proved commodity clusters could process petabytes but wrote to disk between stages. Spark kept data in memory across stages with a resilient DAG, making iterative work an order of magnitude faster, and now spans SQL, streaming and ML in one engine.
Partitions, shuffles and skew
Work parallelises across partitions. Any operation that regroups data by key — joins, group-bys, distinct — triggers a shuffle, which is network- and disk-bound and dominates runtime. Skew, where one key holds most rows, leaves one straggler task; fix with salting, broadcast joins for small tables, or repartitioning.
Choosing an engine
Spark suits large transformations and ML pipelines; Flink is built for true low-latency streaming with event-time and state; Trino/Presto federates interactive SQL over lakes; DuckDB handles single-node analytics astonishingly well. Most datasets are smaller than teams assume — measure before reaching for a cluster.
from pyspark.sql import functions as F
orders = spark.read.format("delta").load("s3://lake/silver/orders")
customers = spark.read.format("delta").load("s3://lake/silver/customers")
daily = (orders
.join(F.broadcast(customers), "customer_id") # small side broadcast
.withColumn("day", F.to_date("created_at"))
.groupBy("day", "country")
.agg(F.sum("total_cents").alias("cents"),
F.countDistinct("customer_id").alias("buyers")))
daily.write.mode("overwrite").partitionBy("day").format("delta").save("s3://lake/gold/daily")