
Ultimate access to all questions.
A Spark application utilizes a broadcast variable to share a small configuration dataset with all executors. The application performs a heavy transformation using this variable in the initial stage, but the variable is not required for the remaining duration of the job. To prevent the broadcast variable from consuming executor memory and causing resource exhaustion in the later stages, what is the most effective method?
A
Ensure the broadcast variable is marked as immutable to prevent inconsistent state across tasks.
B
Increase the spark.sql.broadcastTimeout configuration to allow more time for the variable to be loaded.
C
Manually call the unpersist() method on the broadcast variable immediately after the initial stage is complete.
D
Rely on the Spark Context to automatically garbage collect the variable as soon as the stage utilizing it finishes.