What is the difference between a topic and a partition?

A topic is the named stream that producers write related events into and consumers read from. A partition is a shard of that topic used for scale and parallelism. Ordering is preserved inside a partition, so events that must be handled sequentially should be routed using a stable key.

Why do streaming systems need watermarks?

Watermarks give the processor a practical signal that event time has advanced far enough to close or emit a window. They exist because events can arrive late, out of order, or after retries. Without a watermark policy, windowed results either wait indefinitely or ignore late data too aggressively.

Does exactly-once mean an event is never retried?

No. Exactly-once is about the final effect, not the physical read path. An event may be read again after a crash or retry, but coordinated offsets, state, and sink writes prevent that replay from changing aggregates, tables, or downstream results more than once.

Streaming & Real-Time

05.01 · Concept

Streaming System Concepts

Explain events, topics, partitions, offsets, watermarks, event time, processing time, and exactly-once semantics.

Streaming systems organise immutable events into topics, split them across partitions, and track consumer progress with offsets. Correct processing depends on understanding partition-level ordering, the difference between event time and processing time, how watermarks close windows, and how exactly-once semantics make retries safe rather than impossible.

What this lesson answers

what is an offset in stream processing
event time versus processing time in streaming
how do watermarks handle late events

Notes

A streaming system is built around events: immutable records that describe something that happened, such as an order being placed, a sensor reading being emitted, or a user clicking a button. Events are usually written to a topic, which is a named stream of related data. To scale throughput, a topic is split into partitions, and each event is appended to one partition in order. Within a partition, every event gets an offset, which is a monotonically increasing position that consumers use to track what they have read.

Common questions

What is the difference between a topic and a partition?: A topic is the named stream that producers write related events into and consumers read from. A partition is a shard of that topic used for scale and parallelism. Ordering is preserved inside a partition, so events that must be handled sequentially should be routed using a stable key.
Why do streaming systems need watermarks?: Watermarks give the processor a practical signal that event time has advanced far enough to close or emit a window. They exist because events can arrive late, out of order, or after retries. Without a watermark policy, windowed results either wait indefinitely or ignore late data too aggressively.
Does exactly-once mean an event is never retried?: No. Exactly-once is about the final effect, not the physical read path. An event may be read again after a crash or retry, but coordinated offsets, state, and sink writes prevent that replay from changing aggregates, tables, or downstream results more than once.

Short definition: what is Streaming System Concepts?

04.07 · Batch Processing at ScaleCost and Resource Management 05.02 · Streaming & Real-TimeKafka Producers, Consumers, and Topics