Databricks Certified Data Engineer - Associate
Get started today
Ultimate access to all questions.
A data engineer has three tables in a Delta Live Tables (DLT) pipeline. They have configured the pipeline to drop invalid records at each table. They notice that some data is being dropped due to quality concerns at some point in the DLT pipeline. They would like to determine at which table in their pipeline the data is being dropped. Which of the following approaches can the data engineer take to identify the table that is dropping the records?
Real Exam
Community
KKeng
A
They can set up separate expectations for each table when developing their DLT pipeline.
B
They cannot determine which table is dropping the records.
C
They can set up DIT to notify them via email when records are dropped.
D
They can navigate to the DLT pipeline page, click on each table, and view the data quality statistics.
E
They can navigate to the DIT pipeline page, click on the 'Error' button, and review the present errors.
Explanation:
Explanation
In Delta Live Tables (DLT), data engineers can monitor data quality metrics and identify where records are being dropped through the DLT pipeline UI. Here's why option D is correct:
- DLT Pipeline Monitoring: The DLT pipeline page provides detailed monitoring capabilities for each table in the pipeline.
- Data Quality Statistics: When you click on each table in the DLT pipeline, you can view data quality statistics including:
- Number of records processed
- Number of records dropped due to expectations/constraints
- Quality metrics for each expectation defined on the table
- Table-level Visibility: This approach allows the data engineer to identify exactly which table is dropping records by examining the quality statistics for each table individually.
Why other options are incorrect:
- A: Setting up separate expectations is a development practice, not a monitoring approach to identify where drops are occurring.
- B: This is incorrect because DLT provides monitoring capabilities to track dropped records.
- C: There's no "DIT" (likely a typo for DLT) email notification feature specifically for dropped records in this context.
- E: The "Error" button typically shows pipeline execution errors, not data quality drops at the table level.
Best Practice: To effectively monitor data quality in DLT pipelines, data engineers should:
- Define clear expectations with descriptive names
- Regularly check the DLT pipeline UI for quality metrics
- Use the table-level statistics to identify where data quality issues are occurring
- Consider setting up alerts for critical quality violations if needed
Comments
Loading comments...