Databricks Certified Data Engineer - Associate
Comments
Loading comments...
Ultimate access to all questions.
In your role as a Data Engineer at a company utilizing Azure Databricks for big data analytics, you are tasked with integrating a custom library not available in the default Databricks environment into your project. The library must be compatible with the Databricks runtime, and the solution should minimize maintenance overhead and ensure scalability across multiple clusters. Considering these requirements, which of the following approaches is the BEST to achieve this goal? Choose one option.
A
Manually install the custom library on each node of every cluster, ensuring to check compatibility with the Databricks runtime for each installation.
B
Create a custom Docker image that includes the custom library and its dependencies, then configure all Databricks clusters to use this image as their base, despite the increased complexity in management.
C
Utilize the Databricks library management feature to upload the custom library and its dependencies, then attach it to the necessary clusters, ensuring compatibility and ease of management.
D
Rewrite the custom library's source code to ensure it is fully compatible with the Databricks runtime, then distribute the modified version across all clusters.