Architecture, DataFrames, shuffles, partitioning, tuning
Practise these as flashcards. One or two lines each; the ones you miss come back first.
Start freeThe driver coordinates the execution of Spark jobs, creates the logical plan, and schedules tasks. Executors are worker processes that run the tasks assigned by the driver and handle data processing.
Transformations like map or filter are lazy and build up a logical plan, while actions like count or collect trigger the actual execution of the plan and return results.
Lazy evaluation means Spark doesn't compute results until an action is called. This allows Spark to optimize the execution plan and avoid unnecessary computations.
A job is split into stages based on where shuffles are needed. Each stage consists of tasks, which are units of work for each partition and can run in parallel.
Narrow transformations, like map or filter, only require data from a single partition, so no data movement is needed. Wide transformations, like groupByKey or join, require data from multiple partitions, causing a shuffle across the cluster.
repartition vs coalesce?repartition is used to increase or decrease the number of partitions with a full shuffle, which is useful for balancing data before wide operations. coalesce reduces the number of partitions without a full shuffle, which is more efficient when decreasing partitions after filtering or aggregating.
A shuffle is when data is redistributed across the cluster, usually due to wide transformations like joins or groupBy. Shuffling is expensive because it involves disk and network I/O, and can slow down jobs if not managed carefully.
Data skew happens when some partitions have much more data than others, causing slow tasks. To mitigate skew, I might use salting techniques, increase parallelism, or tune partitioning logic to distribute data more evenly.
I cache or persist a DataFrame when it's reused multiple times in a workflow, especially if it's expensive to recompute. I also consider available memory and choose the appropriate storage level to avoid spilling.
The Spark UI shows job and stage timelines, task durations, failed tasks, data skew, and executor memory usage. This helps identify bottlenecks like slow stages, skewed partitions, or resource constraints.
A broadcast join sends a small table to all executors, avoiding shuffles and speeding up joins with small dimension tables. A sort-merge join shuffles and sorts both tables, which is better for large tables but more resource-intensive.
Structured Streaming treats data as an unbounded table and processes it incrementally, whereas batch processing handles bounded datasets. Structured Streaming supports event-time processing, stateful operations, and continuous output, making it suitable for real-time use cases.