To effectively monitor the performance of your Azure Databricks pipelines over time, identifying trends like increasing run times or memory usage, which approach would you implement for long-term performance monitoring?

Real Exam

Use Databricks' native monitoring tools to manually check job metrics after execution, recording changes over time in a spreadsheet.

3.6%

Set up custom logging in your Spark jobs to gather detailed performance metrics, saving these logs in Azure Blob Storage for regular review.

8.4%

Configure Azure Monitor with Application Insights to gather and analyze performance metrics from Databricks, creating dashboards for trend analysis.

72.3%

Export Databricks metrics to Azure Log Analytics and employ Kusto Query Language (KQL) queries to examine trends and establish alerts for anomalies.

15.7%

Databricks Certified Data Engineer - Professional