Skip to content
Klay

All papers

System Design22 min read

Dremel: Interactive Analysis of Web-Scale Datasets

Sergey Melnik, Andrey Gubarev, Jing Jing Long, et al.2010VLDB 2010

Read it on research.google(opens in a new tab)
  • analytics
  • storage
  • sql
  • columnar

Why this one

Interactive analytics begins when the storage format and execution engine are designed as one machine. Read this after MapReduce: Simplified Data Processing on Large Clusters and The Google File System. MapReduce taught a generation to think in batch jobs, but Dremel explains why people started expecting answers while they were still thinking. The trick is not just columnar storage. It is columnar storage for nested records, plus a serving tree that can fan a query across thousands of machines and combine partial results fast enough to feel interactive. People often reduce this lineage to BigQuery or Parquet trivia, but the useful idea is sharper: if your data model, encoding, and execution tree agree, you can skip most bytes and most coordination. That is the heart of modern analytical systems. Read it before you design an events warehouse, a metrics backend, or anything that promises SQL over logs without admitting where the latency went.

What to take away

  • Dremel shows how repetition and definition levels make nested records work in a columnar format.
  • Its serving tree turns one SQL query into many leaf scans and fast aggregation of partial results.
  • The lesson is byte skipping first, distributed execution second, and only then SQL convenience.

Reads with