Ultimate access to all questions.
Upgrade Now 🚀
Sign in to unlock AI tutor
In the context of a Spark Streaming job that reads from a Kafka source and encounters varying load patterns, what is the most effective method to dynamically manage backpressure to ensure consistent processing rates and prevent system overload?
A
Implement a custom rate limiter within the streaming application to control read rates from Kafka, adjusting based on processing times.
B
Disable backpressure entirely and rely on increasing the number of executors dynamically to handle load spikes.
C
Enable Spark Streaming‘s backpressure mechanism by setting spark.streaming.backpressure.enabled to true, allowing Spark to automatically adjust ingestion rates.
spark.streaming.backpressure.enabled
true
D
Configure a fixed rate limit for ingestion using spark.streaming.kafka.maxRatePerPartition based on peak load observations.
spark.streaming.kafka.maxRatePerPartition
