01.04 · Short
Tooling Landscape for ML Engineers
Compare the roles of Snowflake, BigQuery, Databricks, Kafka, Spark, Airflow, dbt, Feast, Pinecone, and MLflow in a modern data stack.
No video curated for this lesson yet
This lesson is written, ordered and part of the path - the video slot is the only thing still open. We are working through Everything Data lesson by lesson; 55 of 85 have their video so far.
The written notes below cover this idea in full - you lose nothing by reading instead of watching.
Modern data stacks split responsibilities across warehouses, processing engines, stream platforms, orchestrators, transformation frameworks, feature stores, vector databases, and model lifecycle tools. Snowflake, BigQuery, Databricks, Kafka, Spark, Airflow, dbt, Feast, Pinecone, and MLflow each solve a different part of moving data from raw events to production ML systems.
What this lesson answers
- what does each modern data stack tool do
- Snowflake BigQuery Databricks Kafka Spark Airflow dbt differences
- where do Feast Pinecone and MLflow fit
Notes
A modern data stack is usually a collection of specialized tools rather than one system that does everything. Snowflake and BigQuery are cloud data warehouses: they store large amounts of structured data and make it easy to run SQL analytics at scale. Databricks and Spark sit closer to large-scale data processing and machine learning workloads: Spark is the distributed compute engine, while Databricks provides a managed platform around Spark with notebooks, jobs, data lakehouse storage, and ML integrations.
Common questions
- How do Snowflake and BigQuery differ from Databricks and Spark?
- Snowflake and BigQuery are mainly cloud warehouses for storing structured data and running SQL analytics. Spark is a distributed compute engine for large processing jobs and ML workloads. Databricks packages Spark with managed infrastructure, notebooks, scheduled jobs, lakehouse storage, and integrations aimed at data engineering and machine learning teams.
- What are Airflow and dbt used for in a data stack?
- Airflow coordinates work across systems: scheduling jobs, handling dependencies, retrying failures, and chaining tasks together. dbt is narrower and usually runs inside the warehouse, where it turns raw SQL tables into tested, documented, reusable models. Airflow manages pipeline flow, while dbt manages SQL transformation quality.
- Why would an ML engineer need Feast, Pinecone, and MLflow?
- Feast manages reusable model features so training and inference use consistent inputs. Pinecone stores embeddings and supports similarity search for retrieval, recommendations, and semantic lookup. MLflow records experiments, artefacts, parameters, and model versions, giving teams a practical way to compare runs and move models towards deployment.
Short definition: what is Tooling Landscape for ML Engineers?