08.01 · Concept
Data Quality Dimensions
Define measurable checks for freshness, completeness, validity, uniqueness, consistency, accuracy, and distribution stability.
Data quality dimensions convert vague suspicion into concrete checks for arrival time, missing data, allowed values, duplicate records, cross-system agreement, real-world correctness, and distribution drift. Each check needs a metric, threshold, schedule, and failure response matched to the dataset’s operational or analytical impact.
What this lesson answers
- what data quality checks should a pipeline have
- how to measure freshness completeness validity uniqueness
- how to detect distribution drift in datasets
Notes
Data quality dimensions turn vague concerns like “the data looks wrong” into checks you can measure, alert on, and track over time. Freshness asks whether data arrived and was updated when expected, such as verifying that the latest event timestamp is within the last hour or that a daily table loaded before 7 a.m. Completeness checks whether required data is present, for example row counts are within an expected range, required columns are non-null, or every source partition has landed.
Common questions
- What are the main data quality dimensions engineers monitor?
- Common dimensions are freshness, completeness, validity, uniqueness, consistency, accuracy, and distribution stability. Together they cover whether data arrived on time, required fields exist, values are allowed, records are not duplicated, related systems agree, values match reality, and statistical patterns have not shifted unexpectedly.
- How is accuracy different from validity in data quality?
- Validity means a value follows defined rules, such as type, range, format, or allowed category. Accuracy means the value is actually correct when compared with a trusted source or business reality. A postcode can be validly formatted but still belong to the wrong customer address.
- What makes a data quality check useful rather than noisy?
- A useful check has a clear metric, an expected boundary, a sensible run frequency, and a defined response when it fails. It should catch failures that matter to downstream use without alerting on harmless variation. Critical reporting, operational workflows, and model features usually need tighter checks than ad hoc analysis.
Short definition: what is Data Quality Dimensions?