Ensuring data consistency across development, staging, and production environments is crucial for CI/CD pipelines involving data transformation jobs in Azure Databricks. Which advanced approach would you adopt to automate data consistency validation within your deployment process?

Real Exam

Writing custom Spark jobs in Databricks to compare dataset snapshots across environments, triggered as part of the deployment pipeline.

9.5%

Utilizing Azure DevOps to run data validation scripts post-deployment, with manual interventions required to address inconsistencies.

16.2%

Leveraging Azure Data Factory to orchestrate data movement and validation jobs between environments, using checksums for consistency checks.

57.1%

Implementing a blockchain-based ledger within Databricks to record and verify data consistency across environments automatically.

17.1%

Databricks Certified Data Engineer - Professional