Answer-first summary for fast verification
Answer: Utilize the Databricks REST API to programmatically collect and analyze metrics, and integrate with an external monitoring tool like Azure Monitor to raise alerts based on predefined thresholds.
Option B is the best approach because it fully automates the monitoring process by leveraging the Databricks REST API for metric collection and analysis, and integrates with an external monitoring tool like Azure Monitor for alerting. This meets all the given requirements: it reduces manual overhead, ensures timely notifications through external alerting, and is cost-effective as it doesn't require significant changes to existing job codes. Option A involves manual monitoring, which is inefficient and doesn't scale. Option C requires extensive changes to job codes and doesn't automate alerting. Option D, while useful for resource management, doesn't provide the necessary monitoring and alerting capabilities for proactive issue resolution.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
No comments yet.
In a Databricks workspace, you are tasked with implementing a robust and scalable solution to monitor and alert on specific metrics related to cluster performance and job execution. The solution must adhere to the following requirements: 1) Automate the monitoring process to reduce manual overhead, 2) Integrate with an external system for alerting to ensure timely notifications, 3) Be cost-effective without requiring significant changes to existing job codes. Given these constraints, which of the following approaches would be the BEST to achieve these objectives? Please choose one option.
A
Periodically check the metrics through the Databricks UI and manually raise alerts based on observed values, ensuring no additional costs are incurred.
B
Utilize the Databricks REST API to programmatically collect and analyze metrics, and integrate with an external monitoring tool like Azure Monitor to raise alerts based on predefined thresholds.
C
Modify the job code to include extensive custom logging for tracking metrics, and use a separate log analysis tool to monitor these logs for anomalies.
D
Enable autoscaling on the cluster to automatically adjust resources based on workload, assuming it will mitigate all performance-related issues without further monitoring.