How is Parquet different from CSV or JSON?

CSV and JSON are usually row-like text formats: the engine must parse text and often step through fields it does not need. Parquet is binary, typed, columnar, and carries metadata. That lets engines skip unneeded columns, avoid much text parsing, use compression more effectively, and sometimes skip whole row groups using stored statistics.

Does Parquet make every query faster?

No. Parquet helps most when queries read a subset of columns, use filters that match available statistics, and scan enough data for I/O savings to matter. If a query reads almost every column, touches most row groups, or is dominated by joins and computation, the storage format may be a smaller part of the runtime.

What are row groups in Parquet?

A row group is a horizontal slice of the dataset stored inside a Parquet file. For each row group, Parquet stores separate chunks for each column plus metadata about those chunks. Engines use row groups as the unit for skipping, parallel reading, and planning which column data to fetch.

Why does Parquet matter for ML data loading?

Training and feature-generation jobs often need selected feature columns across many examples. Parquet lets the loader read only those feature columns rather than materialising the whole table. That can reduce I/O and memory pressure, but the gain depends on file sizes, row group layout, selected columns, and whether the training pipeline reads data sequentially or repeatedly.

Parquet and Columnar Storage

Parquet is a columnar file format for analytical data, storing values from the same field together with metadata that lets engines read, decompress, and filter only the parts relevant to a query or feature load. It is designed for warehouses, lakehouses, and batch ML workloads where scans usually touch selected columns.

The problem Parquet solves is that analytical workloads rarely behave like application transactions. A dashboard, warehouse query, or training job often needs a few fields from a wide dataset, not every attribute of every record. Text formats and row-oriented layouts make engines parse or scan data that will later be discarded. That wastes I/O, CPU, and memory, especially when tables are wide, nested, or repeatedly read by many jobs.

Parquet stores data by column inside larger blocks called row groups. Within each row group, every column has its own chunk, encoding, compression, and statistics such as value ranges and null information. A query engine first reads the file metadata, decides which row groups and column chunks could matter, then reads only those byte ranges. Similar values sitting together also compress well, so less data has to be fetched and decoded.

Predicate pushdown is the important mechanism people often hand-wave. If a filter cannot match a row group based on its recorded statistics, the engine skips that group before reading the column data. This is not magic indexing, and it is not guaranteed. It depends on useful statistics, filters that line up with stored columns, sensible row group boundaries, and data layout choices such as partitioning, sorting, or clustering.

The trade-off is that Parquet is optimised for scans, not arbitrary small writes or row-by-row access. Tiny files create planning and open-file overhead, while badly sized row groups can hurt parallelism or reduce skipping. Compression saves I/O but costs CPU to decode. Schema evolution is possible, but careless changes still create compatibility problems. The honest answer on performance is always: it depends on file layout, query shape, and engine behaviour.

Engineers meet Parquet in object-store data lakes, warehouse external tables, Spark or DuckDB jobs, feature pipelines, and dataframe exports. In practice, the useful questions are: which columns will readers select, which predicates will they apply, how large should files and row groups be, and whether the data should be partitioned or sorted. For ML, Parquet is especially helpful when loading selected feature columns across many examples.

Go deeper

Everything DataParquet and Columnar StorageExplain how columnar storage, compression, predicate pushdown, and row groups affect query speed and ML data loading.Open the lesson

Common questions

How is Parquet different from CSV or JSON?: CSV and JSON are usually row-like text formats: the engine must parse text and often step through fields it does not need. Parquet is binary, typed, columnar, and carries metadata. That lets engines skip unneeded columns, avoid much text parsing, use compression more effectively, and sometimes skip whole row groups using stored statistics.
Does Parquet make every query faster?: No. Parquet helps most when queries read a subset of columns, use filters that match available statistics, and scan enough data for I/O savings to matter. If a query reads almost every column, touches most row groups, or is dominated by joins and computation, the storage format may be a smaller part of the runtime.
What are row groups in Parquet?: A row group is a horizontal slice of the dataset stored inside a Parquet file. For each row group, Parquet stores separate chunks for each column plus metadata about those chunks. Engines use row groups as the unit for skipping, parallel reading, and planning which column data to fetch.
Why does Parquet matter for ML data loading?: Training and feature-generation jobs often need selected feature columns across many examples. Parquet lets the loader read only those feature columns rather than materialising the whole table. That can reduce I/O and memory pressure, but the gain depends on file sizes, row group layout, selected columns, and whether the training pipeline reads data sequentially or repeatedly.

Every term