06.06 · Concept
Iceberg, Delta Lake, and Table Formats
Compare Iceberg and Delta Lake features including ACID transactions, schema evolution, partition evolution, and time travel.
No video curated for this lesson yet
This lesson is written, ordered and part of the path - the video slot is the only thing still open. We are working through Everything Data lesson by lesson; 55 of 85 have their video so far.
The written notes below cover this idea in full - you lose nothing by reading instead of watching.
Iceberg and Delta Lake turn object storage into reliable analytic tables by tracking table state, committing changes atomically, and exposing historical versions. The practical differences show up in schema changes, partition changes, engine compatibility, retention behaviour, and whether your stack is centred on Databricks or on interoperability across multiple query engines.
What this lesson answers
- Iceberg versus Delta Lake table format differences
- how do ACID transactions work in data lakes
- which table format handles partition evolution better
Notes
Apache Iceberg and Delta Lake are open table formats that add warehouse-like guarantees on top of object storage such as S3, ADLS, or GCS. Instead of treating a data lake as a loose collection of Parquet files, they maintain table metadata that tracks which files belong to the current table version. This enables ACID transactions, so readers see a consistent snapshot while writers commit changes atomically. In practice, this prevents common lake problems such as partially written data, duplicate files from failed jobs, or readers seeing inconsistent results during an overwrite.
Common questions
- Why do Iceberg and Delta Lake exist if Parquet already stores data?
- Parquet stores file contents, not table state. Iceberg and Delta Lake add metadata that says which files make up a valid table version. That gives readers a consistent view while writers update data, and it prevents common object storage failure modes such as incomplete overwrites and orphaned output from failed jobs.
- Is Iceberg better than Delta Lake for schema evolution?
- Iceberg is generally safer for complex schema changes because it tracks columns using stable identifiers rather than relying only on names or positions. Adding columns is routine in both formats, but renames and drops need more care in Delta Lake depending on configuration, reader support, and the surrounding platform.
- How should I choose between Iceberg and Delta Lake?
- Choose based on your execution engines, governance layer, cloud environment, and operational tooling. Delta Lake fits especially well in Databricks-centred stacks with strong managed workflows. Iceberg is attractive when multiple engines need to read and write the same tables with minimal platform lock-in.