Answer-first summary for fast verification
Answer: Leverage the `merge` method to perform an upsert operation with a deletion flag within a single transaction, efficiently handling updates and deletes while minimizing operational overhead and ensuring data integrity.
The BEST approach is to leverage the `merge` method to perform an upsert operation with a deletion flag within a single transaction. This method efficiently handles updates and deletes by utilizing the CDF, ensuring data integrity and minimizing operational overhead. Option A is incorrect because it introduces unnecessary operational overhead by scanning the entire dataset periodically. Option B is incorrect because performing a separate delete operation after the upsert is redundant and increases operational overhead. Option D is incorrect because it bypasses the CDF, risking data integrity and not leveraging Delta Lake's capabilities for efficient change data capture.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
No comments yet.
In a real-time data processing scenario using Delta Lake, you are tasked with efficiently handling updates and deletes by leveraging the Change Data Feed (CDF). The solution must comply with the following constraints: minimize operational overhead, ensure data integrity, and support scalability. Considering these constraints, which of the following approaches BEST utilizes the CDF to achieve these goals? (Choose one option)
A
Implement a batch process that periodically scans the entire dataset to identify changes, marks records for deletion, and then performs a delete operation, ensuring data integrity at the cost of increased operational overhead.
B
Use the merge method to perform an upsert operation with a deletion flag, followed by a separate delete operation to remove the flagged records, balancing operational overhead and data integrity.
C
Leverage the merge method to perform an upsert operation with a deletion flag within a single transaction, efficiently handling updates and deletes while minimizing operational overhead and ensuring data integrity.
D
Directly delete records from the Delta table without using the CDF, relying on external logging for change tracking, which minimizes operational overhead but risks data integrity.