Imagine you are working on a project that requires real-time machine learning predictions. Describe how you would implement a real-time prediction system using Spark ML and what challenges you might encounter in ensuring low latency and high throughput.

Simulated

Implement a batch processing system with Spark ML; challenges include latency due to batch intervals.

13.6%

Use Spark ML's streaming capabilities with low batch intervals; challenges include resource management and data consistency.

86.4%

Develop a standalone scikit-learn model for real-time predictions; challenges include scalability issues.

0.0%

Rely on pre-computed predictions to avoid real-time processing; challenges include outdated predictions.

0.0%

Databricks Certified Machine Learning - Associate