Answer-first summary for fast verification
Answer: MERGE, because it enables you to insert new or updated records into the target table while keeping unchanged records intact, making it ideal for incremental data processing.
The MERGE command is the correct choice because it supports the incremental processing requirement by allowing new or updated records to be inserted into the target table while preserving unchanged records. This approach is both cost-effective and scalable, as it minimizes the amount of data processed and transferred. CREATE OR REPLACE TABLE and INSERT OVERWRITE are not suitable for incremental updates as they replace or overwrite the entire table. COPY INTO, while efficient for loading data, does not support the updating of existing records based on source changes, making it less suitable for this scenario.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
You are designing a data pipeline in Azure Databricks to incrementally process data from a source table to a target table. The pipeline must ensure that only new or updated records are inserted into the target table, while maintaining all existing records that have not changed. Additionally, the solution must be cost-effective and scalable to handle large volumes of data. Which of the following commands should you use to achieve this requirement, and why? (Choose one option.)
A
CREATE OR REPLACE TABLE, because it allows you to create a new table or replace an existing table with new data, ensuring a fresh start for each pipeline run.
B
INSERT OVERWRITE, because it allows you to overwrite the target table with new data from the source, which is efficient for full refreshes but does not support incremental updates.
C
MERGE, because it enables you to insert new or updated records into the target table while keeping unchanged records intact, making it ideal for incremental data processing.
D
COPY INTO, because it efficiently loads data into a table and can skip duplicates, but lacks the ability to update existing records based on changes in the source.
No comments yet.