A data pipeline in Azure Databricks is experiencing intermittent failures that are challenging to diagnose. What steps would you take to enhance logging for better troubleshooting?

Real Exam

Modify the Spark jobs to include custom logging statements that capture detailed execution metrics and state.

10.7%

Use Databricks' built-in event logging with default settings, relying on Azure Monitor for anomaly detection.

17.2%

Enable DEBUG level logging for Spark jobs and configure Databricks to export detailed logs to Azure Log Analytics.

59.0%

Implement a logging sidecar container in Azure Kubernetes Service (AKS) to capture stdout and stderr logs from Databricks jobs.

13.1%

Databricks Certified Data Engineer - Professional