What should I tune first in a slow Spark job?

Start by reading the physical plan and the Spark UI rather than guessing. Find the stage that dominates runtime, then check whether it is caused by shuffle, skew, spill, repeated computation, or poor partitioning. Tune the cause you can see in the metrics, then rerun and compare.

When should I use repartition instead of coalesce?

Use repartition when you need a fresh distribution of data, such as before a wide transformation, to increase parallelism, or to change how output is spread. Use coalesce when you are reducing the number of partitions and want to avoid a full redistribution where possible. Repartition is more flexible, but usually more expensive.

Is caching always a good optimisation in Spark?

No. Cache only when the same DataFrame or RDD is reused and recomputing it is expensive. Cached data consumes executor memory, so excessive caching can trigger spills, evictions, or failures. If a value is used once, or is cheap to rebuild, caching often makes the job worse rather than better.

Why can a broadcast join make Spark faster?

A broadcast join copies the small side of a join to executors, allowing tasks to join locally with partitions of the larger side. That avoids shuffling the small side by join key. It helps only when the broadcasted data reliably fits in executor memory; otherwise it can create memory pressure or failed tasks.

Spark Performance Tuning

Spark performance tuning is the practice of making a Spark application spend less time moving, recomputing, spilling, and waiting on data. It usually means reading the physical plan and runtime metrics, then changing partitioning, shuffle-heavy operations, caching, and join strategy so work is balanced and data movement is avoided where possible.

Spark is built to run work in parallel, but a job can still be slow if the parallelism does not match the data. A few huge partitions leave most executors idle while straggler tasks run. Too many tiny partitions waste time in scheduling and often produce awkward output files. Skew is another common trap: the job looks parallel, but one key or range sends a disproportionate amount of data to a small part of the cluster.

The main mechanism is to change where data sits before expensive operations happen. Repartitioning redistributes records across the cluster, often before a wide transformation or a write. Coalescing reduces partition count with less movement when the data is already being narrowed. For joins, grouping, de-duplication, and ordering, Spark may shuffle data by key or sort order; tuning tries to filter, project, and pre-aggregate before that redistribution.

Broadcasting is a concrete way to avoid part of a join shuffle: Spark sends a small table to each executor, so each task can join locally against its own partition of the large side. Caching is different: it keeps a reused DataFrame or RDD near the executors after an expensive computation. Both are useful only when the size and reuse pattern justify the memory they occupy.

The trade-off is that every tuning knob can make another bottleneck worse. More partitions can improve balance but add overhead. Repartitioning may fix skew or output layout, but it is itself a shuffle. Broadcast joins can fail or create memory pressure when the supposedly small side grows. Caching can save recomputation, or it can evict useful data and cause spills. The honest answer is usually: inspect metrics, change one thing, measure again.

Engineers meet Spark performance tuning in query plans, the Spark UI, job logs, and code reviews around DataFrame transformations. Look for expensive stages, large shuffle reads or writes, spill, long tail tasks, repeated computation, and join choices that do not match the data. A common misunderstanding is that tuning is a bag of magic settings; in practice, most wins come from reducing data movement and making partitions similarly sized.

Go deeper

Everything DataSpark Performance TuningImprove a Spark job by adjusting partitioning, reducing shuffle size, caching selectively, and broadcasting small tables.Open the lesson

Common questions

What should I tune first in a slow Spark job?: Start by reading the physical plan and the Spark UI rather than guessing. Find the stage that dominates runtime, then check whether it is caused by shuffle, skew, spill, repeated computation, or poor partitioning. Tune the cause you can see in the metrics, then rerun and compare.
When should I use repartition instead of coalesce?: Use repartition when you need a fresh distribution of data, such as before a wide transformation, to increase parallelism, or to change how output is spread. Use coalesce when you are reducing the number of partitions and want to avoid a full redistribution where possible. Repartition is more flexible, but usually more expensive.
Is caching always a good optimisation in Spark?: No. Cache only when the same DataFrame or RDD is reused and recomputing it is expensive. Cached data consumes executor memory, so excessive caching can trigger spills, evictions, or failures. If a value is used once, or is cheap to rebuild, caching often makes the job worse rather than better.
Why can a broadcast join make Spark faster?: A broadcast join copies the small side of a join to executors, allowing tasks to join locally with partitions of the larger side. That avoids shuffling the small side by join key. It helps only when the broadcasted data reliably fits in executor memory; otherwise it can create memory pressure or failed tasks.

Every term