What is Ray Data used for in ML pipelines?

Ray Data is used to run dataset preprocessing across a cluster before or alongside training. You define reads, filters, maps, feature extraction, tensor conversion and batching in Python, while Ray schedules the work across available resources and moves data between workers as needed.

How does Ray Data avoid loading the full dataset into memory?

Ray Data represents a dataset as distributed blocks rather than as a single in-memory object. Transformations can be executed over those blocks in parallel, and work can be pipelined where possible, so intermediate results do not always need to be fully materialised before the next stage starts.

Why connect Ray Data directly to distributed training?

Connecting Ray Data to training lets workers consume prepared shards or batches instead of each worker reading and preprocessing the same source data. That separates data preparation from model code while reducing duplicated work, improving throughput, and helping expensive training resources spend less time waiting for input.

Batch Processing at Scale

04.04 · Walkthrough

Ray for Data and ML Workloads

Use Ray Data to preprocess a dataset and feed batches into a distributed training workflow.

Ray Data turns Python preprocessing into a distributed dataset pipeline: read data, transform it in parallel, control partitioning and batching, then stream prepared batches into Ray Train workers. It avoids every trainer repeating the same input work and helps keep distributed training fed without hand-rolling process coordination.

What this lesson answers

how does Ray Data feed Ray Train
Ray Data preprocessing workflow for distributed training
when should I repartition a Ray Dataset

Notes

Ray is a distributed execution framework that lets you scale Python data and machine learning workloads without manually managing worker processes, queues, or cluster coordination. In this lesson, the key idea is that Ray Data provides a dataset abstraction for batch processing across a cluster. Instead of loading all data into one machine’s memory, Ray partitions the data into blocks and executes transformations such as reading, filtering, mapping, shuffling, and batching in parallel.

Common questions

What is Ray Data used for in ML pipelines?: Ray Data is used to run dataset preprocessing across a cluster before or alongside training. You define reads, filters, maps, feature extraction, tensor conversion and batching in Python, while Ray schedules the work across available resources and moves data between workers as needed.
How does Ray Data avoid loading the full dataset into memory?: Ray Data represents a dataset as distributed blocks rather than as a single in-memory object. Transformations can be executed over those blocks in parallel, and work can be pipelined where possible, so intermediate results do not always need to be fully materialised before the next stage starts.
Why connect Ray Data directly to distributed training?: Connecting Ray Data to training lets workers consume prepared shards or batches instead of each worker reading and preprocessing the same source data. That separates data preparation from model code while reducing duplicated work, improving throughput, and helping expensive training resources spend less time waiting for input.

Short definition: what is Ray for Data and ML Workloads?

04.03 · Batch Processing at ScaleSpark Performance Tuning 04.05 · Batch Processing at ScaleBuilding Training Data Jobs