Databricks Certified Data Engineer - Associate

Explanation

In Delta Live Tables (DLT), when you want to identify at which specific table in the pipeline records are being dropped due to quality concerns, the correct approach is:

Option D: Navigate to the DLT pipeline page, click on each table, and view the data quality statistics.

Here's why:

1. Data Quality Statistics in DLT: DLT provides detailed data quality metrics for each table in the pipeline. When you configure expectations with EXPECT or EXPECT ALL and specify ON VIOLATION DROP ROW, DLT tracks:

   • Number of records processed
   • Number of records that passed expectations
   • Number of records that failed expectations
   • Number of records dropped

2. Table-level Monitoring: By clicking on each table in the DLT pipeline UI, you can view:

   • Data Quality tab: Shows detailed statistics about expectations and violations
   • Metrics: Displays counts of dropped records for each expectation
   • Timeline: Shows when records were dropped during pipeline execution

3. Why other options are incorrect:

   • Option A: Setting up separate expectations is necessary for data quality enforcement, but it doesn't help identify which table is dropping records after the pipeline is running.
   • Option B: The "Error" button typically shows pipeline failures or execution errors, not data quality violations where records are dropped.
   • Option C: While DLT can be configured for notifications, email alerts don't provide the granular table-level information needed to identify exactly which table is dropping records.

4. Best Practice: For troubleshooting data quality issues in DLT pipelines:

   • Use descriptive expectation names to easily identify which rules are failing
   • Review data quality statistics after each pipeline update
   • Consider using EXPECT ALL for comprehensive quality monitoring
   • Use the DLT UI's table details to pinpoint exactly where records are being dropped

This approach allows the data engineer to systematically check each table's data quality metrics to identify the specific table(s) where records are being dropped due to quality violations.