Explanation:
Azure Databricks provides built-in support for handling late-arriving data, which is common in real-time data processing scenarios. By specifying a watermark, you can define how late the data can be and still be included in the processing. Additionally, using window operations allows you to manage data that arrives within a certain time frame after the watermark. This approach ensures that the data processing logic can accommodate late data without discarding it or waiting indefinitely.
Ultimate access to all questions.
No comments yet.
In a data processing scenario where you are using Azure Databricks to handle late-arriving data, what strategies would you consider to manage this issue?
A
Set a fixed time window for data processing and discard any data that arrives after the window.
B
Configure the system to wait indefinitely for late-arriving data.
C
Use Azure Databricks' built-in support for handling late data by specifying a watermark and using window operations.
D
Manually track the timestamps of incoming data and filter out any data that is outside the expected time range.