Ultimate access to all questions.
Upgrade Now 🚀
Sign in to unlock AI tutor
How can you design an anomaly detection system in Spark Structured Streaming that adapts thresholds based on machine learning models, enabling real-time model retraining and threshold adjustment?
A
Use foreachBatch to write streaming data to a Delta Lake table, periodically pausing the stream to retrain the model and update thresholds.
B
Implement a continuous learning paradigm where streaming data is used to retrain models on a separate Spark cluster, updating the detection thresholds dynamically.
C
Batch process streaming data periodically, retrain the model offline, and update broadcast variables with new thresholds.
D
Integrate Spark Streaming with an external machine learning service that continuously updates its model and thresholds, querying it for each batch of streaming data.
