Answer: Structured Streaming

Auto Loader utilizes a Structured Streaming source known as `cloudFiles` to automatically process new files as they arrive in a specified cloud storage directory. This mechanism supports the processing of existing files as well. Auto Loader is capable of handling billions of files, facilitating tasks like migration or backfilling a table, and can ingest millions of files per hour in near real-time. Progress tracking is managed through a scalable key-value store (RocksDB) within the pipeline's checkpoint location, ensuring data is processed exactly once. In the event of failures, Auto Loader resumes from the last checkpoint, maintaining fault tolerance and exactly-once semantics without requiring manual state management. Thus, the correct answer is Structured Streaming.

Author: LeetQuiz Editorial Team