Module 04
Batch Processing at Scale
Build scalable batch data processing jobs for large training datasets and offline feature computation.
7 lessons · 3 videos · 2h 27m- 04.01
Distributed Compute Fundamentals
Explain partitions, shuffles, executors, tasks, memory pressure, and data locality in distributed batch processing.
- 04.02
Apache Spark DataFrames
Use Spark DataFrames to read, filter, join, aggregate, and write a large dataset in Parquet format.
- 04.03
Spark Performance Tuning
Improve a Spark job by adjusting partitioning, reducing shuffle size, caching selectively, and broadcasting small tables.
- 04.04
Ray for Data and ML Workloads
Use Ray Data to preprocess a dataset and feed batches into a distributed training workflow.
- 04.05
Building Training Data Jobs
Design a repeatable batch job that creates point-in-time training examples from raw events, labels, and feature tables.
- 04.06
Incremental Batch Processing
Implement an incremental batch job that processes only new partitions and safely updates downstream tables.
- 04.07
Cost and Resource Management
Estimate compute and storage costs for a batch pipeline and identify optimizations that reduce runtime and spend.