The most direct and reliable way to verify predicate push-down is by examining the Physical Plan on the Query Detail screen (under the SQL tab) of the Spark UI.

Key Analysis:

• Physical Plan: In the SQL tab, clicking on a query execution reveals the plan visualization. By inspecting the scan node (e.g., FileScan parquet or BatchScan), you can look for the PushedFilters attribute. If filters are being pushed to the source, they will appear there. If they appear in a separate Filter node above the scan node, it indicates Spark is reading the data first and then filtering it in memory.
• Stage Detail: The "Input" metric shows how much data was read, which can provide an indirect hint (e.g., more data read than expected), but it cannot confirm if the optimizer successfully pushed a specific filter to the storage layer.
• Executor Logs: Optimizer decisions made by the Catalyst Optimizer are not recorded in the executor logs, which are primarily used for task-level debugging, garbage collection, and errors.
• Storage Tab: This tab is dedicated to monitoring cached (persisted) datasets and does not provide information regarding query optimization or filter push-down.
• Delta Transaction Log: While Delta logs contain metadata used for data skipping, they do not show the actual execution plan or Spark's final decision on predicate push-down for a specific query instance.