Databricks Certified Data Engineer - Associate

The correct answer is E. In the Query Detail screen, by interpreting the Physical Plan.

Explanation:

1. Predicate push-down is a query optimization technique where filter conditions (predicates) are pushed down to the data source level to reduce the amount of data read from storage.

2. Spark UI's Query Detail screen provides detailed information about query execution, including the Physical Plan. The Physical Plan shows how Spark will execute the query, including whether filters are applied early (at scan time) or later in the execution pipeline.

3. Diagnosing predicate push-down issues:

   • In the Physical Plan, look for Filter operations that appear after Scan operations rather than being pushed down into the scan.
   • If you see a pattern like Scan → Filter instead of Scan with PushedFilters, it indicates predicate push-down is not being leveraged.
   • The Physical Plan will show the actual execution strategy, allowing you to identify if filters are being applied at the data source level or later in the processing pipeline.

4. Why other options are incorrect:

   • A: Executor logs don't specifically track predicate push-down optimization status.
   • B: The Input column shows data read size, but doesn't indicate whether predicate push-down was used.
   • C: Storage Detail screen shows RDD persistence, not query optimization details.
   • D: Delta Lake transaction log contains metadata and statistics, but doesn't show query execution plans or predicate push-down usage.

5. Key indicators in Physical Plan:

   • Look for PushedFilters in scan operations
   • Check if filters appear immediately after scans
   • Examine the data size before and after filter operations

By analyzing the Physical Plan in the Query Detail screen, you can identify whether predicate push-down optimization is being applied and diagnose performance problems caused by its absence.