Answer-first summary for fast verification
Answer: Using Azure Monitor with Databricks to collect metrics and logs, and view them in Azure Dashboard
Option B is the optimal choice for monitoring a data pipeline in Azure Databricks to pinpoint performance bottlenecks. Azure Monitor offers a comprehensive solution for gathering metrics and logs from Azure services, including Databricks. It facilitates the collection of detailed performance metrics and logs, such as job execution times, resource usage, and error rates. Moreover, Azure Monitor's integration with Azure Dashboard enables the visualization of these metrics and logs in a centralized dashboard, simplifying the analysis process. This approach allows for the swift identification of bottlenecks, troubleshooting, and pipeline optimization. While Option A provides a method for custom logging, it lacks the seamless integration and visualization features of Azure Monitor. Option C offers monitoring and auditing but falls short in delivering detailed performance metrics. Option D, though feasible, lacks real-time monitoring and visualization capabilities, making it less efficient for bottleneck identification.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
To effectively monitor and analyze the performance of a data pipeline in Azure Databricks for identifying bottlenecks, which method should you employ?
A
Implementing custom logging within Databricks notebooks to send metrics to an external monitoring solution
B
Using Azure Monitor with Databricks to collect metrics and logs, and view them in Azure Dashboard
C
Enabling Databricks audit logs and analyzing them in Azure Log Analytics
D
Setting up Databricks to send metrics directly to an Azure Storage account for manual analysis
No comments yet.