How is data lineage different from data versioning?

Data versioning records particular states of data, such as a snapshot of a table or training set. Lineage records relationships between states and processes: which snapshot fed which transformation, which feature set trained which model, and which model served which prediction. Versioning gives stable objects to point at; lineage explains how those objects were connected.

Do you need lineage for every single prediction?

It depends on the product risk and audit requirements. For low-risk aggregate analytics, run-level lineage may be enough. For credit, healthcare, fraud, safety, or customer-facing decisions, teams often need prediction-level traceability. That usually means logging the request, feature values or references, model version, serving code version, and response identifier.

What is training-serving skew in lineage terms?

Training-serving skew is a mismatch between what the model learned from and what it sees in production. Lineage helps expose it by linking the training feature definitions and dataset snapshots to the serving-time feature retrieval and transformation path. If those paths diverge, the lineage graph shows where the model’s training assumptions stopped matching reality.

Training and Serving Data Lineage

Training and serving data lineage is the recorded chain connecting a model output to the data, feature definitions, transformations, code, model artifact, and deployment that produced it. It makes a prediction inspectable as the result of specific upstream states, rather than a standalone value detached from its provenance.

ML systems fail in ways that are hard to diagnose from the prediction alone. A bad score may come from late source data, a changed feature definition, mismatched training and serving logic, a corrupt label, or the wrong model being deployed. Without lineage, teams debug by inference and memory. With lineage, they can ask what exact data and software state existed when the model was trained and when the prediction was served.

Mechanically, lineage is usually modelled as a graph. Nodes are sources, schemas, transformation jobs, feature views, datasets, experiment runs, model artifacts, deployments, requests, and predictions. Edges say that one node was derived from, read by, written by, trained by, or served by another. Useful systems attach immutable identifiers to these nodes, such as dataset versions, commit hashes, job run identifiers, registry versions, image tags, feature view versions, and event times.

The cost is extra engineering discipline and storage. Pipelines, feature stores, registries, orchestrators, and serving services must emit metadata consistently, and that metadata must be queryable after the fact. Lineage also does not magically prove correctness. It can tell you which feature value was used and where it came from, but not whether the feature was conceptually valid. The honest answer to how much lineage is enough depends on risk, regulation, and debugging needs.

Engineers meet this in model registries, experiment tracking, data catalogues, feature stores, workflow orchestrators, online serving logs, and incident reviews. A common misunderstanding is that lineage is only a compliance feature. In practice, it is just as useful for ordinary production work: reproducing a training run, explaining an individual decision, finding training-serving skew, rolling back a deployment, or determining whether a source data change affected existing predictions.

Go deeper

Everything DataTraining and Serving Data LineageTrace a model prediction back to the data sources, feature definitions, transformations, and model version that produced it.Open the lesson

Common questions

How is data lineage different from data versioning?: Data versioning records particular states of data, such as a snapshot of a table or training set. Lineage records relationships between states and processes: which snapshot fed which transformation, which feature set trained which model, and which model served which prediction. Versioning gives stable objects to point at; lineage explains how those objects were connected.
Do you need lineage for every single prediction?: It depends on the product risk and audit requirements. For low-risk aggregate analytics, run-level lineage may be enough. For credit, healthcare, fraud, safety, or customer-facing decisions, teams often need prediction-level traceability. That usually means logging the request, feature values or references, model version, serving code version, and response identifier.
What is training-serving skew in lineage terms?: Training-serving skew is a mismatch between what the model learned from and what it sees in production. Lineage helps expose it by linking the training feature definitions and dataset snapshots to the serving-time feature retrieval and transformation path. If those paths diverge, the lineage graph shows where the model’s training assumptions stopped matching reality.

Every term