For real-time anomaly detection in streaming data using Spark Structured Streaming, which technique enables the dynamic adjustment of detection thresholds based on historical data patterns?

Real Exam

Store historical data in a Delta Lake table, periodically reading it to update a broadcast variable with new thresholds.

4.3%

Implement a static threshold that is manually adjusted on a regular schedule based on historical trends.

7.1%

Utilize a machine learning model within the streaming query to continuously update detection thresholds based on incoming data.

78.6%

Broadcast a dataset containing historical anomalies and use a UDF to compare streaming data against this static dataset.

10.0%

Databricks Certified Data Engineer - Professional