In a distributed computing environment, a data engineer is setting up a streaming data pipeline with Apache Spark Structured Streaming. This pipeline features a stateful operation designed to monitor the running count of events by key. Which configuration is crucial for achieving the best performance and ensuring fault tolerance for this stateful operation?

Real Exam

Broadcasting join tables to all executors to reduce shuffle during state updates.

19.2%

Using a stateful operation that stores state in local memory for faster access.

19.2%

Configuring checkpointing to HDFS to ensure fault tolerance.

34.6%

Disabling write-ahead logs to increase the throughput of the streaming application.

15.4%

Increasing the number of shuffle partitions to maximize parallelism.

11.5%

Databricks Certified Machine Learning - Associate