Databricks Certified Data Engineer - Associate

Explanation

Correct Answer: A (Checkpointing and Write-ahead Logs)

Spark Structured Streaming uses Checkpointing and Write-ahead Logs to reliably track the exact progress of data processing. Here's why:

Key Components:

1. Checkpointing:

   • Stores the offset range of data being processed in each trigger
   • Records the progress of streaming queries
   • Enables fault tolerance by allowing the query to restart from the last checkpoint
   • Typically stored in a reliable file system (like HDFS, S3, or DBFS)

2. Write-ahead Logs (WAL):

   • Records the data that has been received but not yet processed
   • Ensures that no data is lost even if the driver fails
   • Works in conjunction with checkpointing for end-to-end exactly-once semantics

How They Work Together:

• When a trigger occurs, Spark records the offset range of data to be processed
• This information is written to the checkpoint location
• Write-ahead logs ensure that received data is durably stored before processing
• If a failure occurs, Spark can restart from the last checkpoint and reprocess from the recorded offset

Why Other Options Are Incorrect:

• B (Replayable Sources and Idempotent Sinks): While replayable sources and idempotent sinks are important for end-to-end exactly-once processing, they don't directly record the offset range for progress tracking.
• C (Write-ahead Logs and Idempotent Sinks): Idempotent sinks ensure that reprocessing doesn't cause duplicate writes, but they don't track offset ranges.
• D (Checkpointing and Idempotent Sinks): Similar to option C, idempotent sinks handle output deduplication but don't track processing progress.

Key Takeaway:

Checkpointing is specifically designed to record the progress of streaming queries (including offset ranges), while write-ahead logs ensure data durability. Together, they enable Structured Streaming to handle failures reliably by restarting and/or reprocessing from known checkpoints.