
Ultimate access to all questions.
In the context of designing a scalable and maintainable data pipeline for ingesting data from multiple sources into Delta Lake on Microsoft Azure, consider the following scenario: Your organization is expanding its data sources to include streaming data from IoT devices, batch data from on-premises databases, and real-time social media feeds. The pipeline must adhere to strict compliance standards, minimize costs, and be scalable to handle future data growth. Which of the following approaches BEST meets these requirements? Choose one option.
A
Develop a monolithic application that uses a single, fixed method for ingesting all types of data, regardless of source characteristics or volume, to simplify initial development.
B
Create separate, custom ingestion scripts for each data source without utilizing any Azure or open-source data processing frameworks, to ensure complete control over the data flow.
C
Utilize Azure Databricks with Apache Spark and Delta Live Tables to orchestrate and optimize the ingestion process, leveraging built-in scalability, compliance features, and cost-management tools.
D
Focus solely on the immediate data ingestion needs using the simplest tools available, postponing any considerations for scalability, compliance, or cost optimization until the pipeline becomes operational.