06.05 · Concept
Parquet and Columnar Storage
Explain how columnar storage, compression, predicate pushdown, and row groups affect query speed and ML data loading.
Parquet is fast because it stores values by column, compresses similar data together, and exposes metadata that lets engines skip irrelevant work. Row groups, column chunks, statistics, partitioning, and file sizing determine whether queries and ML loaders read only the fields and ranges they need or waste I/O on unused data.
What this lesson answers
- how does Parquet speed up analytical queries
- what are row groups in Parquet files
- why is Parquet useful for ML feature loading
Notes
Parquet is a columnar file format, which means values for each column are stored together instead of storing full rows together. This matters because analytical queries and ML feature loading often need only a subset of columns. If a query reads customer_id, event_time, and purchase_amount from a table with 200 columns, a row-based format may still force the engine to scan through far more data than needed, while Parquet can read only the relevant column chunks.
Common questions
- Why is columnar storage faster for analytics?
- Analytical queries usually touch a limited set of columns across many records. A columnar format lets the engine read those columns directly instead of scanning full rows. It also improves compression because neighbouring values have similar types and patterns, so less data is fetched and less CPU is spent decoding irrelevant fields.
- What does predicate pushdown mean in Parquet?
- Predicate pushdown means the query engine applies filters using file metadata before reading the actual column data. Parquet stores statistics for row groups, such as value ranges and null information. If those statistics prove a row group cannot match the filter, the engine skips it instead of reading and decoding it.
- How does Parquet affect ML data loading?
- ML jobs often need a chosen set of feature columns across many examples. Parquet allows loaders to fetch only those feature columns, reducing I/O, memory pressure, and startup cost. The gains depend on layout: tiny files add planning overhead, while poor row group choices can limit skipping and parallel reading.
Short definition: what is Parquet and Columnar Storage?