Answer-first summary for fast verification
Answer: It scans the input directory to identify new files, loading them incrementally and idempotently into the target Delta Lake table.
The correct answer is **D**. ### Explanation: By default, Databricks Auto Loader operates in **Directory Listing mode**. In this mode, Auto Loader identifies new files by periodically scanning the input directory and tracking the processed files in a checkpoint. This allows for **incremental** and **idempotent** ingestion (ensuring each file is processed exactly once) without requiring complex cloud-native configurations. ### Why the other options are incorrect: * **Option A:** This describes **File Notification mode**. While often more scalable for directories with millions of files, it is not the default and must be explicitly enabled (e.g., by setting `cloudFiles.useNotifications` to `true`). * **Option B:** Auto Loader does not use webhooks to trigger jobs; it is integrated into Spark Structured Streaming and relies on its own discovery mechanisms (listing or notifications). * **Option C:** Auto Loader is designed specifically to avoid full directory scans once it has established a state. It tracks discovered files to ensure that it only processes new data rather than recreating the table from scratch on every run.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
Which statement accurately describes the default execution mode used by Databricks Auto Loader to detect and process new files?
A
It utilizes cloud-specific notification services (e.g., AWS SNS/SQS or Azure Event Grid) and queue storage to track and incrementally load new files.
B
It relies on webhooks to automatically trigger Databricks jobs whenever new files arrive, merging data into target tables using schema inference.
C
It performs a full scan of the source directory during every trigger, recreating the target Delta table by querying all valid files found in the source.
D
It scans the input directory to identify new files, loading them incrementally and idempotently into the target Delta Lake table.
No comments yet.