Answer-first summary for fast verification
Answer: Configure Azure Monitor with Application Insights to gather and analyze performance metrics from Databricks, creating dashboards for trend analysis.
Option C is the most effective method for long-term performance monitoring of Azure Databricks pipelines. By integrating Azure Monitor with Application Insights, you can continuously collect and analyze performance metrics from Databricks. This setup facilitates the tracking of essential performance indicators such as run times and memory usage over time. Dashboards in Azure Monitor offer a straightforward way to visualize trends and spot potential issues. Moreover, configuring alerts for notable changes in performance metrics enables proactive issue resolution. This approach ensures a thorough and streamlined solution for overseeing data pipeline performance over extended periods.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
To effectively monitor the performance of your Azure Databricks pipelines over time, identifying trends like increasing run times or memory usage, which approach would you implement for long-term performance monitoring?
A
Use Databricks' native monitoring tools to manually check job metrics after execution, recording changes over time in a spreadsheet.
B
Set up custom logging in your Spark jobs to gather detailed performance metrics, saving these logs in Azure Blob Storage for regular review.
C
Configure Azure Monitor with Application Insights to gather and analyze performance metrics from Databricks, creating dashboards for trend analysis.
D
Export Databricks metrics to Azure Log Analytics and employ Kusto Query Language (KQL) queries to examine trends and establish alerts for anomalies.
No comments yet.