
Ultimate access to all questions.
In the context of designing a robust data pipeline that processes large volumes of data from multiple sources and writes the results to a Delta table, you are tasked with ensuring atomicity and consistency of data writes, especially in scenarios involving failures. The solution must not only guarantee that all changes are committed or none at all to prevent partial writes but also align with best practices for scalability and cost-efficiency. Considering these requirements, which of the following approaches best leverages Delta Lake's features to achieve atomic and consistent data writes? (Choose one option)
A
Implement a custom checkpointing mechanism outside of Delta Lake to track and manage data writes manually.
B
Rely solely on the durability and high availability of cloud object storage to ensure data consistency without utilizing Delta Lake's transaction log.
C
Use Delta Lake's transaction log to record all changes made to the data, enabling atomic commits and the ability to roll back changes in case of failures.
D
Limit the pipeline's write operations to a single thread or process to avoid concurrency issues, disregarding Delta Lake's built-in features for atomicity.