Primary Indicators of Spark Spilling

Spilling occurs when Spark is forced to move data from memory to disk during a shuffle or sort. You can identify this in two main places:

1. Stage Detail Screen: Navigating to the Stages tab and selecting a specific Stage ID will reveal the Shuffle metrics section. If spilling occurs, two specific columns appear:
   • Shuffle spill (memory): The size of the data in memory before it was spilled.
   • Shuffle spill (disk): The size of the data once serialized and written to disk.
2. Executor Log Files: By accessing the Executors tab and viewing the stdout or stderr logs, you can find explicit log entries from the UnsafeExternalSorter (e.g., Spilling data because...). These entries provide task-level confirmation that memory limits were exceeded.

Why other options are incorrect:

• Query Detail Screen: While useful for viewing the physical and logical plans, the SQL tab does not provide the specific low-level shuffle spill counters found in the Stage detail view.
• Driver Log Files: Spilling is an operation performed by Executors. Consequently, the Driver logs do not record these executor-level memory management events.
• Executor Summary Screen: Although the main Executor tab shows aggregate metrics, it lacks the granular, per-stage shuffle spill metrics necessary for detailed debugging.