Streaming System Concepts
Streaming system concepts are the shared vocabulary for processing unbounded data as it arrives: events, topics, partitions, offsets, time models, watermarks, and consistency guarantees. They describe how records are stored, ordered, replayed, grouped into time windows, and made reliable despite delay, failure, and retries.
The need for these concepts comes from a simple mismatch: real activity does not arrive neatly or all at once. Users go offline, networks retry, producers batch, old data is reloaded, and machines fail mid-write. A streaming job must still know what it has seen, what order matters, which time a calculation should use, and whether replaying data will corrupt the result.
A stream is made of immutable events, usually written to a named topic. A topic is divided into partitions so producers and consumers can work in parallel. Each partition is an append-only log, and each record receives an offset that marks its position. Consumers remember offsets to resume after failure. Ordering is reliable inside one partition, so systems commonly partition by a key when per-customer or per-device order matters.
Time is handled explicitly because arrival time and occurrence time are different things. Event time comes from when the fact happened at the source. Processing time is when the streaming job sees it. Watermarks are the engine’s estimate that event time has advanced far enough to close or emit a window, while lateness rules decide what to do with records that arrive after that point.
The trade-off is that correctness, latency, and cost pull against each other. Waiting longer for late events improves event-time accuracy but delays outputs. More partitions increase parallelism but can make ordering and rebalancing harder. Exactly-once semantics is commonly misunderstood: it does not mean records are never retried. It means offsets, state, and writes are coordinated so repeated processing does not change the final outcome.
Engineers meet these ideas in Kafka-style logs, Flink or Spark Structured Streaming jobs, cloud pub-sub systems, change data capture pipelines, and real-time analytics services. They appear in decisions such as topic design, partition keys, consumer checkpointing, window definitions, watermark policies, idempotent sink writes, and incident recovery. The honest answer to many design questions is: it depends on ordering needs, lateness tolerance, and sink behaviour.
Common questions
- What is the difference between event time and processing time?
- Event time is the timestamp for when the underlying thing happened, usually assigned by the source or producer. Processing time is when the stream processor handles the record. Use event time when results should reflect the real world. Use processing time when the requirement is based on arrival or operational timing.
- Why do partitions matter in a streaming system?
- Partitions let a topic scale by splitting its log into independently consumed pieces. They also define the strongest ordering boundary: records are ordered within a partition, not across the whole topic. If related events must be processed sequentially, choose a partition key that sends those events to the same partition.
- What does a watermark actually do?
- A watermark tells the stream processor that, according to its current estimate, event time has advanced past a point. The engine can then emit or finalise windows whose time range is behind that point. It is not proof that no older event will appear, so late-data handling still matters.
- Does exactly-once mean an event is processed only once internally?
- No. That is the common misunderstanding. After a crash or retry, a system may read or execute work again. Exactly-once semantics means the externally visible effect is applied once, usually by checkpointing offsets with state and making sink writes transactional or idempotent using stable identifiers.