
Ultimate access to all questions.
In a Databricks workspace, you are tasked with implementing a robust and scalable solution to monitor and alert on specific metrics related to cluster performance and job execution. The solution must adhere to the following requirements: 1) Automate the monitoring process to reduce manual overhead, 2) Integrate with an external system for alerting to ensure timely notifications, 3) Be cost-effective without requiring significant changes to existing job codes. Given these constraints, which of the following approaches would be the BEST to achieve these objectives? Please choose one option.
A
Periodically check the metrics through the Databricks UI and manually raise alerts based on observed values, ensuring no additional costs are incurred.
B
Utilize the Databricks REST API to programmatically collect and analyze metrics, and integrate with an external monitoring tool like Azure Monitor to raise alerts based on predefined thresholds.
C
Modify the job code to include extensive custom logging for tracking metrics, and use a separate log analysis tool to monitor these logs for anomalies.
D
Enable autoscaling on the cluster to automatically adjust resources based on workload, assuming it will mitigate all performance-related issues without further monitoring.