Answer-first summary for fast verification
Answer: Combine small tasks into larger tasks to reduce task scheduling overhead.
A high number of small tasks can lead to increased task scheduling overhead, which can impact performance. Combining small tasks into larger tasks can help reduce this overhead and improve the overall efficiency of the stage.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
You are tasked with optimizing the performance of a Spark batch job that processes large datasets. Using the Spark UI, you notice that the event timeline shows a high number of small tasks in a specific stage. What steps would you take to optimize this stage?
A
Increase the number of executors to distribute the load.
B
Combine small tasks into larger tasks to reduce task scheduling overhead.
C
Decrease the memory allocation for executors.
D
Increase the shuffle partition size to reduce the number of tasks.
No comments yet.