
Ultimate access to all questions.
As a data engineer responsible for designing a data architecture on Microsoft Azure that leverages Delta Lake for streaming data ingestion, you are tasked with ensuring the system can handle high-velocity data while maintaining data quality and query performance. Consider the following scenario: Your organization is processing millions of events per second from IoT devices, and the data must be available for real-time analytics with strict compliance and cost constraints. Which of the following approaches BEST addresses these requirements by leveraging Delta Lake's capabilities? Choose one option.
A
Prioritize data ingestion speed over all other considerations, including data quality and query performance, to ensure all data is captured without delay.
B
Implement a completely separate data quality and validation system that operates outside the streaming pipeline, ensuring data integrity but potentially introducing latency.
C
Utilize Delta Lake's micro-batch processing and ACID transactional capabilities to manage high-velocity data ingestion while performing real-time data quality checks within the same pipeline.
D
Disable all Delta Lake features such as indexing and transaction logging to maximize ingestion speed, accepting the trade-off in data quality and query performance.