Databricks Certified Data Engineer - Professional
Get started today
Ultimate access to all questions.
You are tasked with implementing a data pipeline that processes batch data from a financial institution. The data needs to be processed incrementally and deduplicated. Describe the architecture and operations necessary to achieve this using Delta Lake with batch workloads.
Simulated
A
Use a batch processing approach with Delta Lake to handle incremental processing and deduplication.
B
Implement a streaming architecture with Delta Lake using MERGE INTO for deduplication and auto-compaction for incremental processing.
C
Use a combination of Kafka and Delta Lake for batch data without handling deduplication.
D
Implement a custom Python script to handle batch data and deduplication without using Delta Lake.
Comments
Loading comments...