Answer-first summary for fast verification
Answer: The Stage's detail screen and the Executor's log files.
The most reliable indicators of spilling are found in the **Stage's detail screen** and the **Executor's log files**. * **Stage Detail Screen:** This page displays specific columns for **'Shuffle spill (memory)'** and **'Shuffle spill (disk)'**. If these values are non-zero, it indicates that Spark was forced to move data from memory to disk because the allocated memory for the task was insufficient. * **Executor Log Files:** When a task spills, the executor logs (accessible via the Spark UI) record specific messages such as `Spilling UnsafeExternalSorter to disk` or `Task memory spill`. These logs provide granular evidence of the spill at the task level. **Why the other locations are less suitable:** * **Query/Job Screens:** The Job page provides high-level aggregation, and the SQL/Query detail screen often lacks the specific spill metrics required to troubleshoot partition-level issues. * **Driver Logs:** Since spilling is an executor-side event occurring on worker nodes, the Driver logs rarely contain the specific spill warnings. * **Executor Detail Screen:** While the Executors tab shows cumulative spill totals per executor, it does not map the spill to a specific stage or transformation, making it harder to debug the root cause compared to the Stage detail screen.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
No comments yet.
Spill-to-disk incidents frequently occur during the execution of wide transformations. Which two locations within the Spark UI provide the most direct indicators that a partition is spilling to disk?
A
The Driver's log files and the Executor's log files.
B
The Stage's detail screen and the Query's detail screen.
C
The Executor's detail screen and the Executor's log files.
D
The Stage's detail screen and the Executor's log files.
E
The Query's detail screen and the Job's detail screen.