Answer-first summary for fast verification
Answer: Set up Azure Monitor alerts by leveraging specific Spark event logs and metrics that signal performance issues.
The optimal approach to monitor a Spark-based data pipeline in Databricks involves configuring Azure Monitor alerts with specific Spark event logs and metrics. This method ensures proactive and automated monitoring, enabling immediate notification of performance issues. While Databricks provides built-in tools, manual checks may not promptly identify problems. Developing a custom Spark listener is complex and may lack seamless Azure Monitor integration. Workspace auditing, useful for security, is less effective for performance monitoring. Azure Monitor alerts enhance pipeline reliability and efficiency by facilitating quick issue detection and resolution.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
Your team has encountered intermittent performance issues with a Spark-based data pipeline in Databricks. What is the most effective method to monitor and alert on these performance anomalies?
A
Use Databricks' built-in monitoring tools for daily manual checks on job and cluster performance.
B
Enable Databricks workspace auditing and conduct weekly reviews of audit logs for performance degradation indicators.
C
Set up Azure Monitor alerts by leveraging specific Spark event logs and metrics that signal performance issues.
D
Create a custom Spark listener to record performance metrics into Azure Log Analytics and establish anomaly alerts.
No comments yet.