04.01 · Concept
Distributed Compute Fundamentals
Explain partitions, shuffles, executors, tasks, memory pressure, and data locality in distributed batch processing.
No video curated for this lesson yet
This lesson is written, ordered and part of the path - the video slot is the only thing still open. We are working through Everything Data lesson by lesson; 55 of 85 have their video so far.
The written notes below cover this idea in full - you lose nothing by reading instead of watching.
Distributed batch jobs scale by splitting data into balanced partitions, running partition-sized tasks inside executor processes, and avoiding unnecessary movement across the cluster. The hard parts are shuffles, skew, memory pressure, and poor data locality, because they turn parallel work into network-heavy, disk-heavy, or bottlenecked execution.
What this lesson answers
- what is a partition in batch processing
- why are shuffles expensive in distributed jobs
- how do executors tasks and memory relate
Notes
Distributed batch processing works by breaking a large dataset into partitions and processing those partitions in parallel across a cluster. A partition is a slice of the data that can be handled independently for many operations, such as filtering, mapping, or parsing records. The execution engine assigns work as tasks, where each task typically processes one partition. These tasks run inside executors, which are long-lived worker processes on cluster machines that provide CPU, memory, and local storage for the job.
Common questions
- What is the difference between a partition and a task?
- A partition is a chunk of the dataset. A task is the unit of execution that processes work, commonly one partition at a time. The scheduler assigns tasks to executors, which are worker processes with CPU, memory, and local storage. Partition shape strongly affects how evenly tasks run.
- Why do shuffles make batch jobs slower?
- A shuffle redistributes records between executors, usually to bring matching keys together for joins, grouping, distinct operations, or repartitioning. That means intermediate writes, network transfer, and later reads. It also exposes skew, where a hot key sends too much work to one partition.
- What causes memory pressure in distributed batch processing?
- Memory pressure appears when tasks need more working memory than their executor can safely provide. Common causes include large joins, aggregations, wide records, too much task concurrency, and oversized partitions. The engine may spill to disk, slow heavily, or fail with out-of-memory errors.
Short definition: what is Distributed Compute Fundamentals?