Answer-first summary for fast verification
Answer: Add `dbutils.library.installPyPI('custom_lib')` to the cluster's initialization script.
**Correct Answer: D** - **Option A** is incorrect because switching to Databricks Runtime for MLlib does not facilitate the installation of custom Python libraries. - **Option B** is not the recommended method for installing external Python libraries on a Databricks cluster. - **Option C** installs the library only within the context of a single notebook, not across all notebooks on the cluster. - **Option D** is the correct approach. Adding `dbutils.library.installPyPI('custom_lib')` to the cluster's initialization script ensures the library is installed on all nodes, making it available to all notebooks. - **Option E** is incorrect as Databricks supports mechanisms for installing custom Python libraries on clusters. For widespread availability of `custom_lib` across all notebooks on a Databricks cluster, the team should implement **Option D** by incorporating the library installation command into the cluster's initialization script.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
No comments yet.
How can a data science team ensure that a custom Python library, custom_lib, is available to all notebooks running on a shared Databricks cluster?
A
Configure the cluster to use the Databricks Runtime for MLlib.
B
Include the path to custom_lib in the PYTHONPATH variable within the cluster configuration.
C
Execute %pip install custom_lib in any notebook connected to the cluster.
D
Add dbutils.library.installPyPI('custom_lib') to the cluster's initialization script.
E
It is impossible to make custom_lib available on the cluster.