Answer-first summary for fast verification
Answer: Use watermarking to limit the state store size and the `merge` operation to handle late-arriving data, ensuring data consistency.
Watermarking helps in managing the state store size by limiting the amount of state that needs to be maintained for late-arriving data. The `merge` operation in Delta Lake ensures that updates are correctly applied, maintaining data consistency.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
Given a use case where you need to implement a common design pattern using Structured Streaming and Delta Lake, describe how you would set up a streaming query to handle late-arriving data and ensure data consistency. Include the use of watermarking and the merge operation in your answer.
A
Set up a streaming query without using watermarking and directly use the merge operation to handle late-arriving data.
B
Use watermarking to limit the state store size and the merge operation to handle late-arriving data, ensuring data consistency.
C
Ignore late-arriving data and focus only on the current data stream.
D
Use a batch query to handle late-arriving data and avoid using Delta Lake.
No comments yet.