Answer-first summary for fast verification
Answer: Utilize Delta Lake's upsert feature for atomic updates and inserts based on a unique key, and the DELETE command for atomic deletions, ensuring compliance and data integrity without sacrificing performance.
Delta Lake's upsert feature ensures that updates and inserts are performed atomically based on a unique key, which is crucial for maintaining data integrity and complying with GDPR requirements. The DELETE command allows for atomic deletions, ensuring that data is removed consistently. This approach does not compromise on performance or cost, as it leverages Delta Lake's built-in features efficiently. Disabling the transaction log or relying on manual tracking introduces risks of data inconsistency and non-compliance, while automatic compaction does not directly address the need for atomic updates and deletions.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
No comments yet.
In the context of managing a data pipeline that processes sensitive customer information, you are tasked with ensuring that updates and deletions in a Delta table are performed atomically and consistently to maintain data integrity and comply with GDPR requirements. The solution must also optimize for cost and performance. Given these constraints, which of the following approaches best leverages Delta Lake's features to meet these requirements? Choose the best option.
A
Implement a custom solution that manually tracks changes in an external log and applies updates and deletions in batch processes to minimize costs.
B
Disable Delta Lake's transaction log to reduce storage costs and perform direct file operations for updates and deletions, accepting the risk of potential inconsistencies.
C
Utilize Delta Lake's upsert feature for atomic updates and inserts based on a unique key, and the DELETE command for atomic deletions, ensuring compliance and data integrity without sacrificing performance.
D
Rely on Delta Lake's automatic compaction feature to periodically reorganize data files, indirectly managing updates and deletions through file consolidation.