Skip to content
Klay

All papers

System Design18 min read

Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, et al.2012NSDI 2012

Read it on usenix.org(opens in a new tab)
  • spark
  • data
  • fault-tolerance
  • clusters

Why this one

Read this after MapReduce: Simplified Data Processing on Large Clusters and The Google File System. MapReduce made cluster computing survivable by writing between steps. RDDs ask a sharper question: if a dataset was built by deterministic transforms, why store every copy when you can remember how to rebuild lost partitions? That one move, lineage over replication, turns memory from a cache into a programming model and makes iterative jobs, joins, and interactive analysis feel like a different machine. People often reduce Spark to faster Hadoop, but the useful idea is not speed by itself. It is the contract between the scheduler and the data abstraction: coarse transformations are restricted enough to recover cheaply, yet expressive enough for real pipelines. If you build data systems, read it for the shape of the API as much as the runtime.

What to take away

  • Lineage lets Spark recover lost partitions by replaying deterministic transformations instead of eagerly replicating every block.
  • The RDD API is intentionally coarse grained, which gives the scheduler enough structure to optimize placement and recovery.
  • Caching only matters because the model knows when recomputation is cheaper than shipping or storing another copy.

Reads with