Explanation:
To retain manually deleted or updated records in the raw_iot table, the pipelines.reset.allowed property must be set to false on raw_iot (Option A). This prevents the pipeline from resetting the table during updates, preserving manual changes. Downstream tables like bpm_stats will recompute by default during pipeline updates if their own pipelines.reset.allowed property is not explicitly disabled. The skipChangeCommits flag (Options B/D) is unrelated to retaining manual changes and instead handles CDC scenarios by ignoring update/delete events in the source. Setting pipelines.reset.allowed on bpm_stats (Option C) is unnecessary, as the goal is to recompute it, which requires allowing resets (the default behavior).
Ultimate access to all questions.
No comments yet.
How should a data engineer configure a DLT pipeline with the following streaming tables to retain manually deleted or updated records in the raw_iot table (which ingests raw device measurements from a heart rate tracker) while ensuring the downstream bpm_stats table (which computes user statistics from BPM measurements) is recomputed during pipeline updates?
The pipeline consists of:
raw_iot: Ingests raw heart rate device measurementsbpm_stats: Incrementally computes user BPM statistics from raw_iotA
Set the pipelines.reset.allowed property to false on raw_iot
B
Set the skipChangeCommits flag to true on raw_iot
C
Set the pipelines.reset.allowed property to false on bpm_stats
D
Set the skipChangeCommits flag to true on bpm_stats