Explanation:
When the show_orders notebook sets spark.sql.autoBroadcastJoinThreshold to 1024b, it modifies the configuration at the session level for that notebook. Running %run ./show_orders in fetch_results executes this setting within the context of fetch_results, thus the retrieved value reflects the notebook-level override of 1024b.
Ultimate access to all questions.
No comments yet.
A team of data engineers is collaborating on notebooks attached to a shared cluster. The admin initially set the Spark config property spark.sql.autoBroadcastJoinThreshold to 100b during cluster creation. Later, a data engineer in the show_orders notebook changes this property to 1024b using spark.conf.set. Subsequently, another data engineer in the fetch_results notebook runs %run ./show_orders followed by retrieving the property's value. What will be the output of this retrieval?
A
The output will be 1024b because the notebook-level configuration overrides the cluster-level setting.
B
The output will be 100b as the cluster-level configuration cannot be modified by notebook-level settings.
C
The output will be 100b since changes in one notebook do not affect another notebook's configuration.
D
There will be no output due to the absence of a print statement.
E
The output will be None because the property's default value is None.