
Answer-first summary for fast verification
Answer: Delta Live Tables
Delta Live Tables (DLT) is the correct tool for this scenario because it provides built-in data quality monitoring and enforcement capabilities. DLT allows data engineers to define expectations and constraints on data quality, and it can automatically monitor, alert, and even handle data that doesn't meet quality standards. **Key reasons why DLT is the correct choice:** 1. **Data Quality Expectations**: DLT enables you to define expectations on your data using simple declarative syntax 2. **Automated Monitoring**: It automatically tracks data quality metrics and provides visibility into data quality issues 3. **Quality Enforcement**: DLT can be configured to either: - Drop records that fail expectations - Quarantine records for investigation - Halt pipeline execution when quality thresholds are breached 4. **Built-in Quality Dashboard**: Provides visualizations and metrics for data quality monitoring **Why the other options are not correct:** - **Auto Loader**: Primarily for incremental data ingestion from cloud storage, not specifically for data quality monitoring - **Unity Catalog**: Provides data governance, security, and metadata management, but not automated data quality monitoring - **Delta Lake**: Provides ACID transactions and schema enforcement, but lacks the automated quality monitoring and alerting capabilities of DLT DLT's data quality features make it ideal for automating the monitoring of source data quality degradation over time.
Author: Keng Suppaseth
Ultimate access to all questions.
No comments yet.
A data engineer is maintaining a data pipeline. Upon data ingestion, the data engineer notices that the source data is starting to have a lower level of quality. The data engineer would like to automate the process of monitoring the quality level.
Which of the following tools can the data engineer use to solve this problem?
A
Auto Loader
B
Unity Catalog
C
Delta Lake
D
Delta Live Tables