When designing a data quality framework within a lakehouse architecture, which approach ensures comprehensive data validation, error handling, and cleansing without introducing significant processing overhead?

Real Exam

Develop custom Spark jobs that run periodically to validate data quality, applying fixes in batches to reduce impact on primary workloads.

39.8%

Embed data quality checks into streaming ingestion processes, using side outputs to capture and manage errors.

18.0%

Integrate a third-party data quality tool directly into the data ingestion pipeline, performing real-time validation and cleansing.

16.4%

Utilize declarative data quality rules within Delta Lake, enforcing constraints at the point of ingestion.

25.8%

Databricks Certified Data Engineer - Professional