Answer-first summary for fast verification
Answer: Develop a modular data pipeline with distinct processing modules, each optimized for a specific data frequency, allowing for flexible scaling and cost management.
A modular data pipeline with distinct processing modules for each data frequency is the most efficient and scalable approach. It allows for flexible scaling, cost management, and compliance with data governance policies by processing each data frequency optimally. This approach avoids the pitfalls of a one-size-fits-all solution, unnecessary preprocessing costs, and the complexity of managing multiple pipelines.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
As a Microsoft Fabric Analytics Engineer, you are designing a data pipeline to process data from multiple sources, each with varying data frequencies (e.g., real-time, daily, weekly). The solution must be cost-effective, scalable, and comply with data governance policies. Which of the following strategies should you implement to ensure the pipeline processes data efficiently under these constraints? (Choose one correct option)
A
Design a single, monolithic data pipeline that attempts to process all data frequencies uniformly, disregarding the specific requirements of each data source.
B
Develop a modular data pipeline with distinct processing modules, each optimized for a specific data frequency, allowing for flexible scaling and cost management.
C
Preprocess all incoming data to a uniform frequency using dataflows before pipeline processing, potentially increasing latency and processing costs.
D
Create separate data pipelines for each data frequency and manually merge the results, leading to increased complexity and maintenance overhead.
No comments yet.