Databricks Certified Data Engineer - Associate
Get started today
Ultimate access to all questions.
A data engineer is designing a data pipeline. The source system generates files in a shared directory that is also used by other processes. As a result, the files should be kept as is and will accumulate in the directory. The data engineer needs to identify which files are new since the previous run in the pipeline, and set up the pipeline to only ingest those new files with each run.
Which of the following tools can the data engineer use to solve this problem?
Real Exam
Community
KKeng
A
Unity Catalog
B
Delta Lake
C
Databricks SQL
D
Data Explorer
E
Auto Loader
Explanation:
Explanation
Auto Loader is specifically designed for incremental data ingestion scenarios where files accumulate in a directory and you need to process only new files since the last run. Here's why:
Key Features of Auto Loader:
- Incremental File Processing: Auto Loader automatically tracks which files have been processed and only ingests new files in subsequent runs.
- File Notification Mode: Uses cloud-native file notification services (like AWS SQS, Azure Event Grid, or GCP Pub/Sub) to efficiently detect new files without scanning the entire directory.
- Directory Listing Mode: Falls back to directory listing when file notification services aren't available, still providing incremental processing capabilities.
- State Management: Maintains state information about processed files, ensuring exactly-once processing semantics.
Why Other Options Are Incorrect:
- Unity Catalog: A unified governance solution for data and AI assets, not designed for incremental file ingestion.
- Delta Lake: A storage layer that provides ACID transactions and schema enforcement, but doesn't inherently solve the incremental file detection problem.
- Databricks SQL: A SQL analytics service for querying data, not for incremental data ingestion.
- Data Explorer: A tool for exploring and understanding data, not for pipeline orchestration or incremental ingestion.
Use Case Fit:
The described scenario perfectly matches Auto Loader's capabilities:
- Files accumulate in a shared directory
- Need to identify new files since previous pipeline run
- Files should remain in place (not moved/deleted)
- Incremental processing requirement
Auto Loader is the recommended Databricks solution for streaming or incremental file ingestion from cloud storage.
Comments
Loading comments...