Databricks Certified Data Engineer - Associate
Get started today
Ultimate access to all questions.
Comments
Loading comments...
Ultimate access to all questions.
In the context of a data pipeline utilizing Delta Lake for incremental data processing, a scenario arises where a batch of data was incorrectly processed due to a logic error in the transformation code. This error has propagated through several subsequent batches, affecting the integrity of the table. The team needs to revert the table to a state before the error was introduced to correct the data. Considering the need for minimal downtime and ensuring data consistency, which of the following commands in Delta Lake would be the most appropriate to use, and why? Choose the best option from the following:
A
The OPTIMIZE command, as it compacts small files and improves query performance, indirectly helping in data correction by making the data more accessible for manual fixes.
B
The VACUUM command, since it removes unused files and can help in rolling back changes by eliminating files that contain the incorrect data.
C
The DELETE command, allowing the team to manually delete the incorrect data batches and reprocess them correctly.
D
The RESTORE command, because it enables rolling back the table to a specific version before the error was introduced, ensuring data consistency and minimizing downtime by avoiding manual data correction.