Explanation:
Delta Lake automatically versions every commit to a table in its transaction log. Using Time Travel, you can query any historical snapshot by specifying a version number or timestamp. By querying the table 'as of' two successive versions and performing a set-based comparison (such as using the EXCEPT operator or a FULL OUTER JOIN), you can pinpoint the exact rows that changed between versions.
numOutputRows), it does not return the actual data content of the changed records.Ultimate access to all questions.
No comments yet.
A data engineering team updates a Delta Lake table named customer_churn_params nightly by overwriting it with new data. Following each successful update, the team needs to identify the record-level differences between the current version and the previous version. Which method should be used to achieve this?
A
Execute the DESCRIBE HISTORY customer_churn_params command to retrieve the operation metrics and a log of all added or modified records.
B
Analyze the Spark event logs to identify the specific rows that were updated, inserted, or deleted during the job execution.
C
Use Delta Lake’s versioning and time travel features to execute a query (e.g., using EXCEPT or a join) that compares the current version with the previous version.
D
Manually parse the Delta Lake transaction log files (_delta_log) to identify and decode the specific Parquet files containing the new data.