01.07 · Concept
Data Contracts and Governance
Define a data contract that includes schema, freshness, ownership, quality checks, privacy constraints, and downstream ML consumers.
A data contract makes datasets behave like maintained interfaces: producers commit to shape, reliability, ownership, quality, privacy rules, and consumer expectations. It reduces silent breakage by turning schema changes, freshness failures, sensitive fields, and machine learning feature assumptions into explicit, testable obligations.
What this lesson answers
- what should a data contract include
- how do data contracts support data governance
- how should data contracts handle ML consumers
Notes
A data contract is an explicit agreement between the team that produces a dataset and the teams or systems that consume it. In a modern data stack, it plays the same role that an API contract plays in service engineering: it defines what will be delivered, how it will be shaped, how reliable it should be, and who is accountable when something changes or breaks. For a working engineer, the key shift is to stop treating tables, events, and feature datasets as informal byproducts of applications, and instead treat them as stable interfaces.
Common questions
- What is a data contract in a modern data stack?
- A data contract is an agreement between a dataset producer and its consumers. It specifies the expected schema, freshness, quality checks, ownership, privacy constraints, and rules for change. The point is to treat tables, events, and feature sets as stable interfaces rather than accidental outputs of application code.
- What quality checks belong in a data contract?
- Quality checks should be specific enough to automate. Common examples include valid field types, accepted enum values, non-null requirements, unique keys, referential integrity, expected row volumes, completeness checks, and anomaly detection for important metrics. These checks should run in pipelines and feed monitoring and alerting.
- Why include machine learning consumers in a data contract?
- Machine learning systems can fail silently when data changes do not break a pipeline but alter model behaviour. A contract should record feature expectations, stable categories, label availability, point-in-time correctness, and training-serving consistency. Naming those consumers makes data changes part of normal change management instead of hidden model risk.
Short definition: what is Data Contracts and Governance?