When dealing with late-arriving data in a streaming data pipeline using Structured Streaming in Spark on Databricks, what strategy ensures accurate real-time analytics without affecting data integrity or query performance?

Real Exam

Creating a separate processing stream exclusively for late-arriving data, employing upserts to merge this data into the primary analytics tables.

3.5%

Using a two-tier architecture where late data is first stored in a temporary buffer (Delta table) and periodically merged with the main dataset using batch processing.

3.5%

Implementing watermarks to manage late data and windowed aggregations, allowing the system to wait for a specified time for late data before updating the results.

87.7%

Leveraging Apache Kafka alongside Structured Streaming to reprocess data windows when late data is detected, ensuring completeness of data.

5.3%

Databricks Certified Data Engineer - Professional

Get started today

Comments

When dealing with late-arriving data in a streaming data pipeline using Structured Streaming in Spark on Databricks, what strategy ensures accurate real-time analytics without affecting data integrity or query performance?