In a Spark Structured Streaming application with stateful operations, what is the optimal strategy for ensuring efficient fault tolerance through checkpointing while minimizing performance overhead?

Real Exam

Setting the checkpoint interval to a very high value to reduce the frequency of checkpoint writes, trading off recovery time for performance.

9.7%

Configuring checkpointing to HDFS or a cloud-based storage system, carefully selecting the checkpoint interval to balance performance and recovery needs.

81.9%

Disabling checkpointing and relying on Spark‘s in-memory state management to recover from failures.

1.4%

Using local file storage for checkpoint data to speed up read/write operations, despite potential data loss risks.

6.9%

Databricks Certified Data Engineer - Professional

Get started today

Comments

In a Spark Structured Streaming application with stateful operations, what is the optimal strategy for ensuring efficient fault tolerance through checkpointing while minimizing performance overhead?