Answer-first summary for fast verification
Answer: Use the 'High Water Mark' method by setting a variable to track the last processed record and filter the source data accordingly.
Option A is the correct approach as it uses the 'High Water Mark' method, which is an efficient way to handle incremental data loads by tracking the last processed record and filtering the source data accordingly. Option B is not efficient as it loads all data into a staging area, which may result in unnecessary processing and storage costs. Option C requires manual intervention and may not be scalable, while Option D is not suitable as it processes all data in each pipeline run, which is not an incremental load strategy.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
You are tasked with designing a data pipeline that needs to handle incremental data loads from a source system. Which of the following strategies would you use to implement this requirement in Azure Data Factory?
A
Use the 'High Water Mark' method by setting a variable to track the last processed record and filter the source data accordingly.
B
Load all data from the source system into a staging area and then use a 'Filter' activity to select only the new records.
C
Manually update a 'LastProcessedDate' column in a control table to track the last processed records and filter the source data based on this column.
D
Disable the incremental load feature and process all data from the source system in each pipeline run.
No comments yet.