01.05 · Walkthrough
Data Platform Reference Architecture
Draw a reference architecture for an ML data platform covering ingestion, storage, transformation, training data generation, serving, and monitoring.
An ML data platform reference architecture maps the path from operational sources through ingestion, durable storage, transformations, feature and training dataset creation, serving, and observability. It clarifies ownership boundaries, shows where quality and governance controls sit, and gives engineers a shared model for designing, debugging, and extending data flows.
What this lesson answers
- how to design an ML data platform architecture
- what belongs in a data platform reference diagram
- how should training data generation fit data architecture
Notes
A data platform reference architecture is a shared blueprint that shows how data moves from source systems to reliable outputs for analytics and machine learning. For a working engineer, the value is not in memorizing a single “correct” diagram, but in understanding the major responsibilities and boundaries. Typical sources include application databases, event streams, SaaS tools, logs, and external datasets. Ingestion systems bring this data into the platform through batch loads, change data capture, or streaming pipelines.
Common questions
- What should an ML data platform reference architecture include?
- It should show source systems, ingestion paths, persistent storage, transformation jobs, curated datasets, feature or training data generation, serving paths, and monitoring. It should also mark cross-cutting concerns such as orchestration, metadata, access control, lineage, and data quality so teams can see both the data flow and the operational responsibilities.
- Where does training data generation sit in the architecture?
- Training data generation usually sits after raw ingestion and core transformations, but before model training. It takes cleaned and modelled data, creates labels and features, preserves point-in-time correctness, and records lineage. This layer is where teams prevent leakage and keep offline training data aligned with online serving behaviour.
- How is serving different from transformation in a data platform?
- Transformation prepares reliable datasets from raw inputs by cleaning, joining, validating, and modelling data. Serving exposes those prepared outputs to consumers, such as dashboards, APIs, batch prediction jobs, reverse ETL syncs, online feature lookups, or model endpoints. The serving layer is about dependable access patterns, latency needs, and integration contracts.
Short definition: what is Data Platform Reference Architecture?