Answer-first summary for fast verification
Answer: Applying the `dropDuplicates` transformation using a composite key.
To maintain data integrity when moving from Bronze to Silver in a streaming context, the `dropDuplicates` function is used with a composite key. This approach ensures that unique records are identified based on the specified columns and duplicates are discarded before they are written to the Silver table.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
When processing a streaming pipeline from a multiplexed Bronze table to a Silver table, which method is most effective for preventing duplicate records?
A
Implementing a full outer join logic to identify and filter existing records.
B
Applying the dropDuplicates transformation using a composite key.
C
Filtering the stream based solely on a specific event type column.
D
Utilizing an incremental MERGE operation without specifying matching conditions.
No comments yet.