Databricks Certified Data Engineer - Associate
Get started today
Ultimate access to all questions.
Deep dive into the quiz with AI chat providers.
We prepare a focused prompt with your quiz and certificate details so each AI can offer a more tailored, in-depth explanation.
A dataset has been defined using Delta Live Tables and includes an expectations clause:
CONSTRAINT valid_timestamp EXPECT (timestamp > '2020-01-01')
CONSTRAINT valid_timestamp EXPECT (timestamp > '2020-01-01')
What is the expected behavior when a batch of data containing data that violates these constraints is processed?
Exam-Like
Community
KKeng
0
A
Records that violate the expectation are added to the target dataset and recorded as invalid in the event log.
B
Records that violate the expectation are dropped from the target dataset and recorded as invalid in the event log.
C
Records that violate the expectation cause the job to fail.
D
Records that violate the expectation are added to the target dataset and flagged as invalid in a field added to the target dataset.
E
Records that violate the expectation are dropped from the target dataset and loaded into a quarantine table.
Explanation:
Explanation
In Delta Live Tables (DLT), when you define expectations using the CONSTRAINT clause without specifying a retention policy, the default behavior is:
- Records that violate the expectation are dropped from the target dataset
- The violation is recorded in the event log for monitoring and auditing purposes
This is different from:
- Option A: Records are NOT added to the target dataset when they violate constraints
- Option C: The job does NOT fail by default (unless you use
ON VIOLATION FAIL UPDATE) - Option D: Records are NOT added with flags (this would require a different expectation configuration)
- Option E: Records are NOT loaded into a quarantine table (this would require explicit quarantine table configuration)
Key Points:
- DLT expectations help ensure data quality by filtering out invalid records
- The event log tracks metrics about expectation violations
- You can configure different behaviors using expectation policies like
ON VIOLATION FAIL UPDATEorON VIOLATION DROP ROW - The default behavior without explicit policy is to drop violating rows and log the violations
Comments
Loading comments...