A junior data engineer is utilizing the %sh magic command to execute some legacy code. A senior data engineer suggests refactoring the code instead. What could be the reason for avoiding the use of the %sh magic command?

Real Exam

%sh executes shell code exclusively on the local driver machine, leading to significant performance overhead.

75.8%

None of these reasons accurately describe why %sh might need to be avoided.

4.6%

%sh restarts the Python interpreter, clearing all variables declared in the notebook.

5.9%

%sh cannot access storage to persist the output.

8.5%

All the above reasons explain why %sh may need to be avoided.

5.2%

Databricks Certified Data Engineer - Professional