Explanation:
To ensure data integrity when moving from bronze to silver in a streaming pipeline, using dropDuplicates() with a composite key is the standard practice. This approach ensures that unique records are identified based on a combination of columns, allowing the engine to filter out duplicates before they are committed to the silver table.
Ultimate access to all questions.
No comments yet.
Which architectural approach is most effective for preventing duplicate records when streaming data from a multiplexed bronze table into a silver table?
A
Filtering the incoming stream based on an event type column.
B
Applying a full outer join to identify and discard existing records.
C
Employing the dropDuplicates function with a defined composite key.
D
Implementing an incremental MERGE statement without specifying a matching condition.