4 · Data engineering

12. Streaming Data and Real-Time Analytics

Kafka, event time, windows, watermarks and exactly-once semantics.

10 min read · 3 MCQs

The log abstraction

Kafka stores an append-only, partitioned, replayable log. Producers append, consumer groups read at their own offsets, and retention decouples fast producers from slow consumers. Ordering holds within a partition only, so the partition key determines what stays ordered.

Event time and windows

Processing time is when you saw the event; event time is when it happened. Real analytics uses event time with tumbling, sliding or session windows, plus watermarks that declare how long to wait for stragglers before closing a window. Allowed lateness trades completeness against latency.

Delivery guarantees

At-least-once plus idempotent sinks is the pragmatic default. Exactly-once needs transactional writes and checkpointed operator state, as Flink and Kafka Streams provide. Always run a batch reconciliation job — streaming state is fragile and drifts.

Chapter quiz

3 questions · pass mark 75%
  1. 1. Kafka guarantees ordering…

  2. 2. A watermark declares…

  3. 3. The practical default for stream processing is…

Answer every question to submit. Progress for da-12 is saved in this browser.