Databricks Certified Data Engineer - Professional
Get started today
Ultimate access to all questions.
Comments
Loading comments...
Ultimate access to all questions.
When utilizing watermarking within a Spark Structured Streaming pipeline integrated with Delta Lake, how is late-arriving data managed?
A
The system guarantees that all late-arriving data is processed by dynamically extending the state duration.
B
Data arriving within the specified watermark threshold is processed, while data arriving outside this threshold is discarded.
C
Late-arriving data is automatically rerouted to a separate Delta table for manual reconciliation.
D
Watermarking ensures that late-arriving data is re-sorted into the correct historical partitions without affecting the current state.