Explanation:
Configuring each service to send logs to a centralized Azure Log Analytics workspace and using Kusto Query Language (KQL) for cross-service analysis is the most suitable approach for ensuring comprehensive monitoring and logging across all the services in an end-to-end data pipeline. This method provides centralized visibility, powerful querying capabilities, real-time monitoring, scalability, and efficiency, making it the ideal choice for monitoring complex data pipelines in Azure.
Ultimate access to all questions.
To achieve comprehensive monitoring and logging across an end-to-end data pipeline that spans multiple Azure services such as Azure Data Factory, Azure Databricks, and Azure Synapse Analytics, what is the most effective approach?
A
Developing custom logging within each component of the pipeline and storing logs in Azure Blob Storage for manual review
B
Using Azure Monitor‘s built-in integration with each service for individual monitoring, without central log aggregation
C
Configuring each service to send logs to a centralized Azure Log Analytics workspace and using Kusto Query Language (KQL) for cross-service analysis
D
Implementing Azure Event Grid to trigger alerts based on log events from each service, analyzed in Azure Functions for cross-service insights
No comments yet.