PrepPilotPractise free

Apache Spark interview questions

Architecture, DataFrames, shuffles, partitioning, tuning

17 questions · PySpark · updated November 2026

Practise these as flashcards. One or two lines each; the ones you miss come back first.

Start free

Core questions

What is the role of the Spark driver and executors?

The driver coordinates the execution of Spark jobs, creates the logical plan, and schedules tasks. Executors are worker processes that run the tasks assigned by the driver and handle data processing.

What is the difference between a transformation and an action in Spark?

Transformations like map or filter are lazy and build up a logical plan, while actions like count or collect trigger the actual execution of the plan and return results.

What is lazy evaluation in Spark and why is it important?

Lazy evaluation means Spark doesn't compute results until an action is called. This allows Spark to optimize the execution plan and avoid unnecessary computations.

How does Spark split a job into stages and tasks?

A job is split into stages based on where shuffles are needed. Each stage consists of tasks, which are units of work for each partition and can run in parallel.

Intermediate questions

Explain the difference between narrow and wide transformations with examples.

Narrow transformations, like map or filter, only require data from a single partition, so no data movement is needed. Wide transformations, like groupByKey or join, require data from multiple partitions, causing a shuffle across the cluster.

When would you use repartition vs coalesce?

repartition is used to increase or decrease the number of partitions with a full shuffle, which is useful for balancing data before wide operations. coalesce reduces the number of partitions without a full shuffle, which is more efficient when decreasing partitions after filtering or aggregating.

What is a shuffle in Spark and why can it be expensive?

A shuffle is when data is redistributed across the cluster, usually due to wide transformations like joins or groupBy. Shuffling is expensive because it involves disk and network I/O, and can slow down jobs if not managed carefully.

How does Spark handle data skew and what can you do to mitigate it?

Data skew happens when some partitions have much more data than others, causing slow tasks. To mitigate skew, I might use salting techniques, increase parallelism, or tune partitioning logic to distribute data more evenly.

How do you decide when to cache or persist a DataFrame?

I cache or persist a DataFrame when it's reused multiple times in a workflow, especially if it's expensive to recompute. I also consider available memory and choose the appropriate storage level to avoid spilling.

What information can you find in the Spark UI to help debug a slow job?

The Spark UI shows job and stage timelines, task durations, failed tasks, data skew, and executor memory usage. This helps identify bottlenecks like slow stages, skewed partitions, or resource constraints.

What is the difference between a broadcast join and a sort-merge join in Spark?

A broadcast join sends a small table to all executors, avoiding shuffles and speeding up joins with small dimension tables. A sort-merge join shuffles and sorts both tables, which is better for large tables but more resource-intensive.

How does Spark Structured Streaming differ from batch processing?

Structured Streaming treats data as an unbounded table and processes it incrementally, whereas batch processing handles bounded datasets. Structured Streaming supports event-time processing, stateful operations, and continuous output, making it suitable for real-time use cases.

What is the default threshold for automatic broadcast joins in Spark, and how can you override it?

See the answer in the app, free

Senior questions

How would you debug a Spark job that is running much slower than expected?

See the answer in the app, free

Describe strategies to handle skewed joins in Spark.

See the answer in the app, free

How do you optimize Spark jobs for both performance and resource usage?

See the answer in the app, free

How would you design a robust Structured Streaming pipeline in Spark?

See the answer in the app, free

More Data Engineering topics