Ultimate access to all questions.
Upgrade Now 🚀
Sign in to unlock AI tutor
To effectively monitor the performance of your Azure Databricks pipelines over time, identifying trends like increasing run times or memory usage, which approach would you implement for long-term performance monitoring?
A
Use Databricks' native monitoring tools to manually check job metrics after execution, recording changes over time in a spreadsheet.
B
Set up custom logging in your Spark jobs to gather detailed performance metrics, saving these logs in Azure Blob Storage for regular review.
C
Configure Azure Monitor with Application Insights to gather and analyze performance metrics from Databricks, creating dashboards for trend analysis.
D
Export Databricks metrics to Azure Log Analytics and employ Kusto Query Language (KQL) queries to examine trends and establish alerts for anomalies.
