Databricks Certified Data Engineer - Professional
Comments
Loading comments...
Ultimate access to all questions.
In the context of designing a data pipeline that ingests data from multiple sources into a Delta table, you are tasked with ensuring data consistency and preventing data conflicts. The solution must adhere to the following constraints: it should be scalable to handle high volumes of data, maintain high performance during data ingestion, and ensure compliance with data governance policies. Considering these requirements, which of the following approaches best leverages Delta Lake's features to meet these objectives? Choose the best option.
A
Implementing external data validation checks before ingestion to ensure data quality, but this may introduce latency and does not directly address concurrent write conflicts.
B
Utilizing Delta Lake's Optimistic Transactional Control (OTC) feature, which provides a lock-free mechanism to manage concurrent writes, ensuring data consistency and conflict resolution without sacrificing performance.
C
Disabling the transaction log to minimize overhead and accelerate data ingestion, though this compromises data consistency and conflict resolution capabilities.
D
Relying solely on Delta Lake's schema enforcement feature to prevent schema mismatches, which addresses only a subset of potential data conflicts and does not ensure consistency in concurrent writes.