Answer-first summary for fast verification
Answer: Incorporate schema evolution controls within the pipeline's configuration to enable automatic adaptation to schema changes.
The correct answer is **B. Incorporate schema evolution controls within the pipeline's configuration to enable automatic adaptation to schema changes.** This method is pivotal for a DLT pipeline processing streaming data as it allows the pipeline to dynamically adjust to schema changes in the upstream data without manual intervention, thereby preserving data integrity. This approach is superior to manual adjustments (option D) as it eliminates downtime and human error. While Delta Lake's mergeSchema option (option C) offers a solution, embedding schema evolution controls directly into the pipeline configuration provides greater flexibility and reliability. Although a preliminary schema detection notebook (option A) can aid in identifying necessary changes, integrating schema evolution controls is a more proactive and automated strategy for real-time schema adaptation.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
In the context of a DLT pipeline handling streaming data, what approach ensures the pipeline can automatically adapt to upstream schema changes without compromising data integrity?
A
Utilize a preliminary schema detection notebook that executes before the pipeline to pinpoint and implement required modifications.
B
Incorporate schema evolution controls within the pipeline's configuration to enable automatic adaptation to schema changes.
C
Depend on Delta Lake's mergeSchema feature within the pipeline to manage schema alterations.
D
Manually halt the pipeline, adapt to schema changes, and then resume processing.
No comments yet.