Curation in progress
Everything Data
Twelve modules from ingestion to the warehouse: SQL for engineers, modelling, batch and streaming pipelines, orchestration, quality, vector databases and the RAG pipeline, versioning, feature stores, and preparing data for a foundation model. All 85 lessons are written; the curation is still filling in behind them, and each lesson says where it stands.
- Modules
- 12
- Lessons
- 85
- Watch time
- 26h 36m
- Free to open
- 4
The Modern Data Stack
Understand how modern data platforms are organized for analytics, machine learning, real-time systems, and AI applications.
- 01.01From Data Warehouse to AI PlatformExplain how warehouses, lakes, lakehouses, streaming systems, feature stores, and vector databases fit into an end-to-end AI data platform.
- 01.02Data Mesh and Domain OwnershipMap a centralized data platform into domain-owned data products with clear ownership, contracts, and consumers.
- 01.03Lakehouse Architecture OverviewIdentify when to use object storage, table formats, compute engines, catalogs, and warehouses in a lakehouse architecture.
- 01.04Tooling Landscape for ML EngineersCompare the roles of Snowflake, BigQuery, Databricks, Kafka, Spark, Airflow, dbt, Feast, Pinecone, and MLflow in a modern data stack.
- 01.05Data Platform Reference ArchitectureDraw a reference architecture for an ML data platform covering ingestion, storage, transformation, training data generation, serving, and monitoring.
- 01.06Batch, Stream, and Serving BoundariesDecide whether a data use case should be implemented with batch processing, streaming processing, online serving, or a hybrid architecture.
- 01.07Data Contracts and GovernanceDefine a data contract that includes schema, freshness, ownership, quality checks, privacy constraints, and downstream ML consumers.
SQL for ML Engineers
Use SQL to create reliable datasets, features, labels, and diagnostics for machine learning workflows.
- 02.01SQL for Training DatasetsWrite SQL queries that assemble feature columns, labels, entity identifiers, and event timestamps into a training dataset.
- 02.02CTEs for Readable TransformationsRefactor a nested SQL query into named CTEs that separate filtering, joins, aggregations, and final feature selection.
- 02.03Window Functions for FeaturesUse window functions to compute rolling counts, lag features, rank features, and time-bounded aggregates for ML examples.
- 02.04Point-in-Time Correct JoinsImplement an as-of join that prevents future data leakage when joining historical features to labels.
- 02.05Query Plans and OptimizationRead a query execution plan and improve a slow query by reducing scanned data, optimizing joins, and using partition filters.
- 02.06dbt Models for ML TransformationsCreate dbt models that materialize staging, intermediate, and feature tables with tests and documentation.
- 02.07SQL Testing and AssertionsWrite SQL-based assertions that validate uniqueness, null rates, accepted values, freshness, and label distribution constraints.
Data Modeling
Design data models that support analytics, feature engineering, training, and serving without leakage or excessive complexity.
- 03.01Entities, Events, and ObservationsDefine entities, events, observations, labels, and timestamps for a supervised ML problem.
- 03.02Star Schemas for Analytics and MLDesign a star schema with fact tables and dimension tables that can support both BI queries and feature generation.
- 03.03Normalization vs Denormalization for MLChoose between normalized and denormalized representations based on update frequency, query patterns, training cost, and leakage risk.
- 03.04Slowly Changing DimensionsModel changing attributes using Type 1 and Type 2 slowly changing dimensions for point-in-time feature correctness.
- 03.05Feature Store Data ModelsDefine feature views, entities, feature values, timestamps, and online keys for a feature store.
- 03.06Label Modeling and LeakageCreate a label table with observation windows and prediction windows that avoids target leakage.
- 03.07Modeling for Multi-Tenant ProductsAdd tenant boundaries, access controls, and partitioning strategies to a data model used by multiple customers.
Batch Processing at Scale
Build scalable batch data processing jobs for large training datasets and offline feature computation.
- 04.01Distributed Compute FundamentalsExplain partitions, shuffles, executors, tasks, memory pressure, and data locality in distributed batch processing.
- 04.02Apache Spark DataFramesUse Spark DataFrames to read, filter, join, aggregate, and write a large dataset in Parquet format.
- 04.03Spark Performance TuningImprove a Spark job by adjusting partitioning, reducing shuffle size, caching selectively, and broadcasting small tables.
- 04.04Ray for Data and ML WorkloadsUse Ray Data to preprocess a dataset and feed batches into a distributed training workflow.
- 04.05Building Training Data JobsDesign a repeatable batch job that creates point-in-time training examples from raw events, labels, and feature tables.
- 04.06Incremental Batch ProcessingImplement an incremental batch job that processes only new partitions and safely updates downstream tables.
- 04.07Cost and Resource ManagementEstimate compute and storage costs for a batch pipeline and identify optimizations that reduce runtime and spend.
Streaming & Real-Time
Design and implement streaming data systems for low-latency features, real-time analytics, and event-driven ML applications.
- 05.01Streaming System ConceptsExplain events, topics, partitions, offsets, watermarks, event time, processing time, and exactly-once semantics.
- 05.02Kafka Producers, Consumers, and TopicsCreate a Kafka topic, publish structured events, consume them, and inspect offsets and consumer group behavior.
- 05.03Schema Registry and Event ContractsDefine an event schema with compatibility rules that prevents breaking downstream streaming consumers.
- 05.04Flink Stream ProcessingBuild a Flink job that computes windowed aggregates with event-time handling and late-arriving data support.
- 05.05Streaming Features for MLDesign streaming feature computations for counters, rolling windows, session attributes, and real-time personalization.
- 05.06Change Data CaptureUse a CDC pattern to capture database inserts, updates, and deletes and publish them into a streaming pipeline.
- 05.07Backpressure, Replay, and Failure RecoveryDiagnose lag, backpressure, poison messages, and replay requirements in a streaming architecture.
Warehouses & Lakehouses
Use modern warehouses and lakehouse technologies to store, query, optimize, and govern data for AI systems.
- 06.01Warehouse vs Lakehouse TradeoffsChoose between a cloud warehouse, lakehouse, or hybrid architecture for specific ML and analytics workloads.
- 06.02Snowflake for ML DataCreate tables, load data, run transformations, and manage compute warehouses for an ML training dataset in Snowflake.
- 06.03BigQuery for Large-Scale AnalyticsPartition, cluster, query, and cost-estimate a large ML feature table in BigQuery.
- 06.04Databricks and Unified AnalyticsUse Databricks notebooks and Delta tables to build an offline feature transformation pipeline.
- 06.05Parquet and Columnar StorageExplain how columnar storage, compression, predicate pushdown, and row groups affect query speed and ML data loading.
- 06.06Iceberg, Delta Lake, and Table FormatsCompare Iceberg and Delta Lake features including ACID transactions, schema evolution, partition evolution, and time travel.
- 06.07Catalogs, Lineage, and Access ControlConfigure a conceptual data catalog entry with ownership, schema metadata, lineage, tags, and access policies.
Orchestration
Build reliable, observable, and maintainable data pipelines using modern workflow orchestration patterns.
- 07.01Workflow Orchestration FundamentalsDescribe DAGs, tasks, dependencies, schedules, sensors, retries, backfills, SLAs, and idempotency.
- 07.02Airflow DAG AuthoringCreate an Airflow DAG that runs an extract, transform, validate, and publish workflow on a schedule.
- 07.03Dagster Software-Defined AssetsModel a pipeline in Dagster as assets with dependencies, partitions, metadata, and materialization checks.
- 07.04Prefect Flows and DeploymentsBuild and deploy a Prefect flow with task retries, parameters, logging, and a scheduled run.
- 07.05Pipeline Design PatternsApply fan-out, fan-in, dynamic mapping, partitioned execution, staging, and checkpointing patterns to a data pipeline.
- 07.06Retries, Idempotency, and BackfillsDesign a pipeline task that can be retried or backfilled without producing duplicate or inconsistent outputs.
- 07.07Orchestration for ML TrainingOrchestrate a training-data generation, model training, evaluation, and model registration workflow with clear dependencies.
Data Quality & Monitoring
Validate, monitor, and debug data and ML pipelines before bad data reaches models or users.
- 08.01Data Quality DimensionsDefine measurable checks for freshness, completeness, validity, uniqueness, consistency, accuracy, and distribution stability.
- 08.02Great Expectations ValidationCreate a Great Expectations suite that validates schema, null rates, ranges, categorical values, and row counts.
- 08.03Contract Tests for PipelinesWrite producer-consumer data contract tests that catch breaking schema and semantic changes before deployment.
- 08.04Data Drift DetectionCalculate feature drift metrics such as PSI, KL divergence, and distribution summary changes on production data.
- 08.05Label and Concept DriftDistinguish feature drift, label drift, and concept drift and select monitoring signals for each.
- 08.06ML Observability DashboardsDesign a dashboard that tracks data freshness, feature distributions, prediction distributions, latency, and model performance.
- 08.07Incident Response for Data PipelinesCreate a runbook for triaging a data quality incident from alert to rollback, replay, or downstream notification.
Vector Databases & RAG Pipelines
Build retrieval systems that transform documents into embeddings, store them in vector databases, and serve them to LLM applications.
- 09.01Embeddings and Vector SearchExplain embedding vectors, similarity metrics, nearest neighbor search, recall, precision, and ranking in retrieval systems.
- 09.02Document Ingestion and ChunkingBuild a document ingestion step that extracts text, chunks documents, attaches metadata, and prepares records for embedding.
- 09.03Embedding Pipeline DesignDesign an embedding pipeline with batching, retry handling, model versioning, metadata capture, and incremental updates.
- 09.04Pinecone Managed Vector SearchCreate a Pinecone index, upsert embeddings with metadata, run similarity queries, and filter search results.
- 09.05pgvector in PostgresStore embeddings in Postgres with pgvector and execute vector similarity queries with metadata constraints.
- 09.06Milvus and Open-Source Vector SearchCompare Milvus collection design, indexing options, and deployment tradeoffs against managed vector databases.
- 09.07RAG Evaluation and Retrieval QualityEvaluate a RAG retriever using recall at k, mean reciprocal rank, groundedness checks, and qualitative failure analysis.
Data Versioning & ML Reproducibility
Make datasets, features, experiments, and model outputs reproducible across time, teams, and environments.
- 10.01The Reproducibility Problem in MLIdentify the data, code, environment, configuration, randomness, and infrastructure variables required to reproduce an ML run.
- 10.02Dataset Versioning PrinciplesDefine a versioning strategy for raw data, transformed data, labels, features, and training splits.
- 10.03DVC for Data and PipelinesTrack a dataset and preprocessing pipeline with DVC and reproduce the same training dataset from versioned artifacts.
- 10.04LakeFS for Data Lake BranchesUse a LakeFS-style branching workflow to isolate, validate, and merge changes to data lake tables.
- 10.05Experiment Tracking MetadataLog parameters, metrics, artifacts, dataset versions, model versions, and environment metadata for an ML experiment.
- 10.06Training and Serving Data LineageTrace a model prediction back to the data sources, feature definitions, transformations, and model version that produced it.
- 10.07Reproducible Training SplitsCreate deterministic train, validation, and test splits that remain stable as new data arrives.
Feature Engineering & Stores
Design, compute, store, serve, and monitor features for offline training and online inference.
- 11.01Feature Engineering LifecycleDescribe the lifecycle of a feature from definition and computation to validation, serving, monitoring, and deprecation.
- 11.02Offline vs Online FeaturesDetermine whether a feature should be computed offline, online, on demand, or through a hybrid batch-stream pipeline.
- 11.03Feast Feature Store WalkthroughDefine Feast entities, feature views, data sources, and retrieval logic for both training and online inference.
- 11.04Tecton and Managed Feature PlatformsCompare managed feature platform capabilities including transformation management, online serving, monitoring, and governance.
- 11.05Point-in-Time Feature RetrievalGenerate a point-in-time correct training dataset from historical feature values and labeled events.
- 11.06Online Serving and Low-Latency AccessDesign an online feature serving path using entity keys, freshness constraints, cache behavior, and latency budgets.
- 11.07Feature Reuse, Discovery, and GovernanceCreate feature metadata that enables discovery, ownership, reuse, quality tracking, and safe deprecation.
Data for LLMs & Foundation Models
Build data pipelines for pretraining, fine-tuning, evaluation, synthetic data generation, and human feedback loops for foundation models.
- 12.01The LLM Data LifecycleDescribe the data stages for pretraining, instruction tuning, preference tuning, evaluation, retrieval augmentation, and production feedback.
- 12.02Data Curation for Foundation ModelsDefine a curation pipeline that filters documents by quality, language, source, license, safety, and domain relevance.
- 12.03Deduplication and Contamination ControlApply exact, fuzzy, and embedding-based deduplication techniques and check for evaluation set contamination.
- 12.04Tokenization and Dataset PackingTokenize text, estimate token distributions, and pack examples into fixed-length sequences for efficient training.
- 12.05Instruction Tuning DatasetsDesign an instruction tuning dataset with prompts, responses, task categories, quality labels, and train-validation-test splits.
- 12.06Synthetic Data Generation PipelinesBuild a synthetic data generation workflow with prompt templates, sampling controls, validation filters, and provenance tracking.
- 12.07RLHF and Preference Data PipelinesModel preference data with prompts, candidate responses, rankings, annotator metadata, quality controls, and reward-model training splits.
- 12.08LLM Evaluation Data ManagementCreate an evaluation dataset registry with task definitions, versioned prompts, expected outputs, rubrics, and leakage controls.