Databricks Certified Data Engineer - Associate

Explanation

In Databricks, there are several ways to install Python packages, but the question specifically asks for a method that installs packages scoped at the notebook level to all nodes in the currently active cluster.

Let's analyze each option:

A. Run source env/bin/activate in a notebook setup script

• This activates a virtual environment but doesn't install packages
• Not a method for installing packages

B. Use b in a notebook cell

• This appears to be incomplete or incorrect syntax
• There's no standard b command for installing packages in Databricks

C. Use %pip install in a notebook cell ✓ CORRECT

• The %pip install magic command installs packages at the notebook level
• When run in a notebook cell, it installs packages to all nodes in the currently active cluster
• This is the recommended method for notebook-scoped installations

D. Use %sh pip install in a notebook cell

• This uses the %sh magic command to run shell commands
• While it can install packages, it only installs them on the driver node, not all worker nodes
• Not scoped to all nodes in the cluster

E. Install libraries from PyPI using the cluster UI

• This installs packages at the cluster level, not notebook level
• These packages are available to all notebooks using the cluster
• Not scoped at the notebook level as requested

Key Points:

• %pip install is the magic command specifically designed for notebook-scoped package installation
• It ensures packages are installed on all nodes (driver and workers) in the active cluster
• The installation is temporary and only persists for the duration of the notebook session
• This is different from cluster-level installations which are permanent and affect all notebooks using that cluster