Databricks Certified Data Engineer - Associate
Get started today
Ultimate access to all questions.
Ultimate access to all questions.
A data engineering team has built a pipeline using Auto Loader to ingest data from cloud storage, processing it every hour. Initially, the source data was partitioned by year, month, and day. Recently, a new partition column, hour, was added to the directory structure of new data files, causing the pipeline to fail for a few hours. The team has now addressed this issue in the code. What should the team do next to ingest and reprocess the data that was not loaded due to this issue?
Explanation:
Auto Loader can resume from its last checkpoint in case of failures, thanks to the information stored in the checkpoint location. This feature ensures exactly-once guarantees when writing data into Delta Lake, eliminating the need for manual state management to achieve fault tolerance or exactly-once semantics. Therefore, the correct answer is: 'Auto Loader uses checkpointing and write-ahead logs to allow a terminated stream to be restarted and continue from where it left off to ensure end-to-end, exactly-once semantics under any failure condition.'