Explanation:
Incremental processing focuses on identifying and processing only the records that have changed or been added since the last execution. By leveraging metadata like timestamps or version columns, data engineers can efficiently filter for new data, avoiding the high cost and overhead of full table scans and full table rewrites.
Ultimate access to all questions.
Which of the following describes a valid approach for implementing incremental processing during the transition from a Bronze to a Silver layer within a Medallion architecture?
A
Rewriting the entire Silver table using a complete overwrite for every incoming batch.
B
Filtering for new or modified records by utilizing metadata such as timestamps or version columns.
C
Performing a full table scan on the Bronze table to ensure data consistency for every batch.
D
Utilizing watermarks exclusively to process late-arriving data while ignoring standard updates.
No comments yet.