04.05 · Concept
Building Training Data Jobs
Design a repeatable batch job that creates point-in-time training examples from raw events, labels, and feature tables.
No video curated for this lesson yet
This lesson is written, ordered and part of the path - the video slot is the only thing still open. We are working through Everything Data lesson by lesson; 55 of 85 have their video so far.
The written notes below cover this idea in full - you lose nothing by reading instead of watching.
Point-in-time training data is built by anchoring each example to an entity and prediction timestamp, then attaching only labels from after that time and features known no later than that time. A reliable batch job makes eligibility, windows, joins, versions, validation, and reruns explicit.
What this lesson answers
- how to build point in time training data
- how to prevent leakage in training data jobs
- what should a training data batch job validate
Notes
A training data job turns raw operational data into a versioned dataset that can be used to train and evaluate a model. The key idea is to construct examples as they would have looked at a specific point in time: each row should represent an entity, a prediction timestamp, the label observed after that timestamp, and the features that were available at or before that timestamp. This prevents data leakage, where the model accidentally learns from information that would not have been known in production.
Common questions
- What is point-in-time training data?
- Point-in-time training data represents what the system would have known at the moment a prediction was made. Each example has an entity, a prediction time, future-derived label, and historical features. The important constraint is that features must not include information created after the prediction timestamp.
- Why do training data jobs cause leakage?
- Leakage usually appears when joins ignore time. A common failure is taking the latest feature value or status for an entity, even if that value was created after the prediction time. The model then learns from information that would not exist in production, making offline results look better than real performance.
- What makes a training data job repeatable?
- A repeatable job produces the same dataset for the same input snapshot and parameters. It records source versions, code version, time windows, feature definitions, and output partitions. It also validates counts, nulls, duplicates, label balance, and feature freshness before the dataset is passed to model training.