Answer: Checkpointing and Write-ahead Logs

## Explanation **Correct Answer: A (Checkpointing and Write-ahead Logs)** Spark Structured Streaming uses **Checkpointing and Write-ahead Logs** to reliably track the exact progress of data processing. Here's why: ### Key Components: 1. **Checkpointing**: - Stores the offset range of data being processed in each trigger - Records the progress of streaming queries - Enables fault tolerance by allowing the query to restart from the last checkpoint - Typically stored in a reliable file system (like HDFS, S3, or DBFS) 2. **Write-ahead Logs (WAL)**: - Records the data that has been received but not yet processed - Ensures that no data is lost even if the driver fails - Works in conjunction with checkpointing for end-to-end exactly-once semantics ### How They Work Together: - When a trigger occurs, Spark records the offset range of data to be processed - This information is written to the checkpoint location - Write-ahead logs ensure that received data is durably stored before processing - If a failure occurs, Spark can restart from the last checkpoint and reprocess from the recorded offset ### Why Other Options Are Incorrect: - **B (Replayable Sources and Idempotent Sinks)**: While replayable sources and idempotent sinks are important for end-to-end exactly-once processing, they don't directly record the offset range for progress tracking. - **C (Write-ahead Logs and Idempotent Sinks)**: Idempotent sinks ensure that reprocessing doesn't cause duplicate writes, but they don't track offset ranges. - **D (Checkpointing and Idempotent Sinks)**: Similar to option C, idempotent sinks handle output deduplication but don't track processing progress. ### Key Takeaway: Checkpointing is specifically designed to record the progress of streaming queries (including offset ranges), while write-ahead logs ensure data durability. Together, they enable Structured Streaming to handle failures reliably by restarting and/or reprocessing from known checkpoints.

Author: Keng Suppaseth