The most reliable indicators of spilling are found in the Stage's detail screen and the Executor's log files.

• Stage Detail Screen: This page displays specific columns for 'Shuffle spill (memory)' and 'Shuffle spill (disk)'. If these values are non-zero, it indicates that Spark was forced to move data from memory to disk because the allocated memory for the task was insufficient.
• Executor Log Files: When a task spills, the executor logs (accessible via the Spark UI) record specific messages such as Spilling UnsafeExternalSorter to disk or Task memory spill. These logs provide granular evidence of the spill at the task level.

Why the other locations are less suitable:

• Query/Job Screens: The Job page provides high-level aggregation, and the SQL/Query detail screen often lacks the specific spill metrics required to troubleshoot partition-level issues.
• Driver Logs: Since spilling is an executor-side event occurring on worker nodes, the Driver logs rarely contain the specific spill warnings.
• Executor Detail Screen: While the Executors tab shows cumulative spill totals per executor, it does not map the spill to a specific stage or transformation, making it harder to debug the root cause compared to the Stage detail screen.