Explanation:
Incremental processing is a core optimization in data engineering. By focusing strictly on records that have been added or updated since the previous batch (often utilizing tools like Auto Loader or Delta Lake's change data feed), engineers can significantly reduce computational overhead and improve pipeline performance compared to full-refresh patterns.
Ultimate access to all questions.
No comments yet.
In a Medallion architecture, what is the primary objective of employing incremental processing when transitioning data from the Bronze layer to the Silver layer?
A
To overwrite the target Silver table with a fresh snapshot of the source data during every execution batch.
B
To perform a full historical scan of all records within the Bronze layer to ensure audit compliance.
C
To process only newly arrived or modified records, thereby optimizing resource consumption and reducing latency.
D
To trigger a full re-computation of all downstream Gold tables regardless of whether the source data has changed.