Databricks Certified Data Engineer - Associate
Get started today
Ultimate access to all questions.
A data engineer has three tables in a Delta Live Tables (DLT) pipeline. They have configured the pipeline to drop invalid records at each table. They notice that some data is being dropped due to quality concerns at some point in the DLT pipeline. They would like to determine at which table in their pipeline the data is being dropped.
Which of the following approaches can the data engineer take to identify the table that is dropping the records?
Real Exam
Community
KKeng
A
They can set up separate expectations for each table when developing their DLT pipeline.
B
They cannot determine which table is dropping the records.
C
They can set up DLT to notify them via email when records are dropped.
D
They can navigate to the DLT pipeline page, click on each table, and view the data quality statistics.
E
They can navigate to the DLT pipeline page, click on the "Error" button, and review the present errors.
Explanation:
Explanation
In Delta Live Tables (DLT), data quality statistics are tracked for each table when expectations are configured. When records are dropped due to quality concerns, DLT provides detailed metrics showing:
- Number of records processed
- Number of records that passed expectations
- Number of records that failed expectations
- Number of records dropped
By navigating to the DLT pipeline page and clicking on each table, the data engineer can view these data quality statistics to identify which table is dropping records.
Let's analyze each option:
- A: Setting up separate expectations for each table is necessary for tracking quality, but this alone doesn't help identify which table is dropping records - it just enables the tracking.
- B: This is incorrect because DLT does provide mechanisms to identify where records are being dropped.
- C: Email notifications can alert about dropped records but don't specifically identify which table is responsible.
- D: CORRECT - This approach directly allows the engineer to see data quality statistics per table, including dropped records.
- E: The "Error" button typically shows pipeline execution errors, not data quality statistics for dropped records.
Key Insight: DLT's data quality dashboard provides per-table statistics that show exactly how many records were dropped at each table due to failed expectations, making option D the most direct and effective approach.
Comments
Loading comments...