01.01 · Concept · Free
From Data Warehouse to AI Platform
Explain how warehouses, lakes, lakehouses, streaming systems, feature stores, and vector databases fit into an end-to-end AI data platform.
Curated for this lesson
The Modern Data Stack
Data Warehouse vs Data Lake vs Data Lakehouse | ETL, OLAP vs OLTP
A concise conceptual foundation for warehouses, lakes, and lakehouses before extending to AI platform components.
An AI data platform is a set of specialised data layers: warehouses for governed business facts, lakes and lakehouses for flexible storage and processing, streams for fresh events, feature stores for consistent model inputs, and vector databases for embedding search. The architecture connects these systems with shared governance, metadata, orchestration, and data contracts.
What this lesson answers
- how do data warehouses fit into AI platforms
- data lake versus lakehouse for machine learning
- what are feature stores and vector databases for
Notes
A modern AI data platform is best understood as an evolution of the traditional data warehouse, not a replacement for it. The warehouse remains the system of record for curated, governed, structured business data: facts, dimensions, metrics, and reporting models. It is where teams define trusted revenue numbers, customer segments, product usage metrics, and operational KPIs. For AI use cases, this curated data is still essential because models need reliable historical examples, labels, and context. The limitation is that warehouses are usually optimized for analytical SQL workloads, not for storing every raw file, processing unstructured data, serving low-latency features, or handling real-time event streams.
That is where the rest of the platform fits in. A data lake stores raw and semi-structured data cheaply and flexibly, such as logs, JSON events, documents, images, and exports from operational systems. A lakehouse adds warehouse-like structure, transactions, governance, and query performance on top of the lake, making it easier to use the same data for analytics, machine learning, and batch processing. Streaming systems move events through the platform in near real time, allowing teams to react to user behavior, transactions, sensor readings, or application logs as they happen. In practice, batch pipelines and streaming pipelines often coexist: batch pipelines build historical training sets and aggregate metrics, while streaming pipelines update current state and power time-sensitive applications.
AI-specific systems sit on top of these foundations. A feature store manages reusable model inputs, keeping the offline training version of a feature consistent with the online serving version used in production. This matters because a model trained on one definition of “last 30 days of activity” should not be served with a slightly different definition in the application. A vector database stores embeddings for text, images, products, users, or other entities so applications can perform semantic search, recommendations, retrieval-augmented generation, and similarity matching. End to end, the platform turns raw data into trusted data, trusted data into features and embeddings, and those assets into models and AI-powered products. The key engineering idea is that no single storage system does everything well; the modern stack is a set of specialized layers connected by governance, metadata, orchestration, and clear data contracts.
Common questions
- Is a data lakehouse a replacement for a data warehouse?
- Not usually. A warehouse remains the best place for curated business metrics, governed reporting models, and trusted structured data. A lakehouse extends the platform by bringing stronger structure and queryability to lake data, especially when analytics, machine learning, and batch processing need to work over broader raw and semi-structured sources.
- Why does an AI platform need both batch and streaming pipelines?
- Batch pipelines are well suited to building historical datasets, labels, aggregates, and training inputs. Streaming pipelines handle events as they arrive, which matters for current user state, fraud signals, operational alerts, and time-sensitive product behaviour. Most production AI platforms need both because training and serving have different freshness requirements.
- What problem does a feature store solve?
- A feature store keeps model inputs reusable, governed, and consistent between training and production serving. Without one, teams often recreate the same feature logic in different places, causing subtle mismatches. That can make a model learn from one definition of a signal and then receive a different definition at runtime.
Short definition: what is From Data Warehouse to AI Platform?