Answer-first summary for fast verification
Answer: Employing the `dropDuplicates` function with a defined composite key.
To ensure data integrity when moving from bronze to silver in a streaming pipeline, using `dropDuplicates()` with a composite key is the standard practice. This approach ensures that unique records are identified based on a combination of columns, allowing the engine to filter out duplicates before they are committed to the silver table.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
No comments yet.
Which architectural approach is most effective for preventing duplicate records when streaming data from a multiplexed bronze table into a silver table?
A
Filtering the incoming stream based on an event type column.
B
Applying a full outer join to identify and discard existing records.
C
Employing the dropDuplicates function with a defined composite key.
D
Implementing an incremental MERGE statement without specifying a matching condition.