Your team has encountered intermittent performance issues with a Spark-based data pipeline in Databricks. What is the most effective method to monitor and alert on these performance anomalies?

Real Exam

Use Databricks' built-in monitoring tools for daily manual checks on job and cluster performance.

1.9%

Enable Databricks workspace auditing and conduct weekly reviews of audit logs for performance degradation indicators.

11.1%

Set up Azure Monitor alerts by leveraging specific Spark event logs and metrics that signal performance issues.

79.6%

Create a custom Spark listener to record performance metrics into Azure Log Analytics and establish anomaly alerts.

7.4%

Databricks Certified Data Engineer - Professional