How does an incremental batch job know what to process?

It compares source data with durable state from previous successful runs. That state might come from a control table, existing target partitions, ingestion metadata, or source and destination watermarks. The important rule is that progress is recorded only after the downstream write succeeds, so retries do not skip work.

What makes an incremental batch job safe to rerun?

The job must be idempotent for each partition. A rerun for the same input should leave the destination in the same final state, not append duplicate rows. Common approaches include writing to staging first, validating the result, then replacing a partition or performing a keyed merge into the target table.

How should late-arriving or corrected source data be handled?

The pipeline needs an explicit backfill path and clear rules for mutable records. Late data may require reprocessing affected partitions, while updates and deletes usually need keyed merge behaviour rather than simple append. Operationally, keep processing history, row counts, and quality checks so corrected ranges can be identified and republished safely.

Batch Processing at Scale

04.06 · Walkthrough

Incremental Batch Processing

Implement an incremental batch job that processes only new partitions and safely updates downstream tables.

No video curated for this lesson yet

This lesson is written, ordered and part of the path - the video slot is the only thing still open. We are working through Everything Data lesson by lesson; 55 of 85 have their video so far.

The written notes below cover this idea in full - you lose nothing by reading instead of watching.

Incremental batch processing runs a batch job only for data that is new or changed since the last successful run. In partitioned lakes and warehouses, that means discovering unprocessed partitions, transforming just those partitions, and publishing results in a way that can be retried safely without duplication or skipped data.

What this lesson answers

how to process only new partitions in batch jobs
how to make incremental batch jobs idempotent
how to safely update downstream partitioned tables

Notes

Incremental batch processing is the practice of running a batch job over only the data that has arrived or changed since the last successful run, instead of reprocessing the entire dataset every time. In a partitioned data lake or warehouse, this usually means detecting new partitions, such as new dates, hours, regions, or ingestion batches, and processing only those partitions. This keeps jobs faster, cheaper, and easier to operate at scale.

Common questions

How does an incremental batch job know what to process?: It compares source data with durable state from previous successful runs. That state might come from a control table, existing target partitions, ingestion metadata, or source and destination watermarks. The important rule is that progress is recorded only after the downstream write succeeds, so retries do not skip work.
What makes an incremental batch job safe to rerun?: The job must be idempotent for each partition. A rerun for the same input should leave the destination in the same final state, not append duplicate rows. Common approaches include writing to staging first, validating the result, then replacing a partition or performing a keyed merge into the target table.
How should late-arriving or corrected source data be handled?: The pipeline needs an explicit backfill path and clear rules for mutable records. Late data may require reprocessing affected partitions, while updates and deletes usually need keyed merge behaviour rather than simple append. Operationally, keep processing history, row counts, and quality checks so corrected ranges can be identified and republished safely.

Short definition: what is Incremental Batch Processing?

04.05 · Batch Processing at ScaleBuilding Training Data Jobs 04.07 · Batch Processing at ScaleCost and Resource Management