Explanation:
To maintain data integrity when moving from Bronze to Silver in a streaming context, the dropDuplicates function is used with a composite key. This approach ensures that unique records are identified based on the specified columns and duplicates are discarded before they are written to the Silver table.
Ultimate access to all questions.
When processing a streaming pipeline from a multiplexed Bronze table to a Silver table, which method is most effective for preventing duplicate records?
A
Implementing a full outer join logic to identify and filter existing records.
B
Applying the dropDuplicates transformation using a composite key.
C
Filtering the stream based solely on a specific event type column.
D
Utilizing an incremental MERGE operation without specifying matching conditions.
No comments yet.