05.04 · Walkthrough
Flink Stream Processing
Build a Flink job that computes windowed aggregates with event-time handling and late-arriving data support.
Flink windowed aggregation depends on event time, watermarks, and an explicit lateness policy. Jobs keep state for open windows, emit results when event-time progress passes a boundary, and can revise or route late records depending on how much correctness, freshness, and retained state the pipeline can afford.
What this lesson answers
- how does Flink handle event time windows
- what are watermarks in Flink stream processing
- how to handle late events in Flink
Notes
Flink is designed for stateful stream processing, where each incoming record can update long-running computations such as counts, sums, averages, or more complex metrics. In this lesson, the key idea is that a Flink job continuously consumes events, assigns them to logical windows, and emits aggregate results for each window. A window might represent “orders per customer every five minutes” or “error counts per service every one minute.” Unlike a batch job, the input is unbounded, so Flink must decide when a window is complete enough to produce a result while still keeping the job running…
Common questions
- Why use event time instead of processing time in Flink?
- Event time is based on when the event actually occurred, rather than when a Flink task happened to receive it. That matters when records arrive out of order because of retries, queues, mobile clients, or network delay. It lets windowed results reflect the source timeline instead of the ingestion timeline.
- What does a watermark do in a Flink job?
- A watermark is Flink’s signal that event time has advanced far enough to close or evaluate earlier windows. When the watermark moves past a window boundary, Flink can run the window aggregate and emit a result, while still allowing behaviour to be configured for records that arrive after that point.
- What happens to events that arrive too late?
- Late events can update an already emitted window if allowed lateness is configured. If they arrive beyond that tolerance, they are usually not folded into the main aggregate. A common pattern is to send them to a side output so they can be inspected, replayed, or handled by a separate correction path.
Short definition: what is Flink Stream Processing?