Ultimate access to all questions.
Upgrade Now 🚀
Sign in to unlock AI tutor
For real-time anomaly detection in streaming data using Spark Structured Streaming, which technique enables the dynamic adjustment of detection thresholds based on historical data patterns?
A
Store historical data in a Delta Lake table, periodically reading it to update a broadcast variable with new thresholds.
B
Implement a static threshold that is manually adjusted on a regular schedule based on historical trends.
C
Utilize a machine learning model within the streaming query to continuously update detection thresholds based on incoming data.
D
Broadcast a dataset containing historical anomalies and use a UDF to compare streaming data against this static dataset.
