Answer-first summary for fast verification
Answer: Leveraging timestamps or version columns to filter for and process only new or updated records.
Incremental processing is designed to optimize performance by focusing only on data that has changed or been added since the last run. By using metadata like **timestamps or version columns**, the engine can filter the source data effectively, avoiding the overhead of full table scans and minimizing the resource consumption associated with full rewrites.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
In a medallion architecture, which technique exemplifies applying incremental processing during the transition from the Bronze to the Silver layer?
A
Performing a full table scan on the Bronze table to identify change sets.
B
Utilizing a watermark to exclusively process data that arrives after a specific latency threshold.
C
Completely rewriting the target Silver table during every processing batch.
D
Leveraging timestamps or version columns to filter for and process only new or updated records.
No comments yet.