Answer-first summary for fast verification
Answer: The output will be `1024b` because the notebook-level configuration overrides the cluster-level setting.
When the `show_orders` notebook sets `spark.sql.autoBroadcastJoinThreshold` to `1024b`, it modifies the configuration at the session level for that notebook. Running `%run ./show_orders` in `fetch_results` executes this setting within the context of `fetch_results`, thus the retrieved value reflects the notebook-level override of `1024b`.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
No comments yet.
A team of data engineers is collaborating on notebooks attached to a shared cluster. The admin initially set the Spark config property spark.sql.autoBroadcastJoinThreshold to 100b during cluster creation. Later, a data engineer in the show_orders notebook changes this property to 1024b using spark.conf.set. Subsequently, another data engineer in the fetch_results notebook runs %run ./show_orders followed by retrieving the property's value. What will be the output of this retrieval?
A
The output will be 1024b because the notebook-level configuration overrides the cluster-level setting.
B
The output will be 100b as the cluster-level configuration cannot be modified by notebook-level settings.
C
The output will be 100b since changes in one notebook do not affect another notebook's configuration.
D
There will be no output due to the absence of a print statement.
E
The output will be None because the property's default value is None.