Ultimate access to all questions.
Upgrade Now 🚀
Sign in to unlock AI tutor
What is the recommended approach for loading small datasets (~10MB or less) in Hyperopt with SparkTrials, and why?
A
Save the dataset to DBFS and load it back onto workers using the DBFS local file interface.
B
Load the dataset on the driver and call it directly from the objective function.
C
Broadcast the dataset explicitly using Spark and load it back onto workers using the broadcasted variable in the objective function.
D
Use Databricks Runtime 6.4 ML or above for efficient handling of small datasets.
