Answer-first summary for fast verification
Answer: Manually select a Spark version that is documented to be compatible with the specific library in question, and configure the cluster to use this version, ensuring to review the Databricks runtime release notes for any known issues.
Option B is the correct approach because it directly addresses the need for compatibility with the specific Spark library by selecting a known compatible Spark version. This method also considers the project's constraints by avoiding the unpredictability of the latest version (Option A), the potential incompatibility of the default version (Option C), and the unnecessary complexity and cost of a custom build (Option D). Reviewing the Databricks runtime release notes ensures that any known issues with the selected version are accounted for, aligning with best practices for configuration and compliance.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
No comments yet.
As a Databricks Certified Data Engineer, you are tasked with configuring a Databricks cluster to use a specific Spark version to ensure compatibility with a particular Spark library. The project has strict compliance requirements and must adhere to cost constraints while ensuring high scalability. Given these constraints, which of the following steps would be the BEST approach to achieve this, and why? (Choose one option)
A
Automatically select the latest Spark version available in Databricks for all clusters to ensure you always have the newest features and compatibility, regardless of specific library requirements.
B
Manually select a Spark version that is documented to be compatible with the specific library in question, and configure the cluster to use this version, ensuring to review the Databricks runtime release notes for any known issues.
C
Use the default Spark version provided by Databricks without any modifications, assuming that the default settings will meet all project requirements, including library compatibility.
D
Build a custom Spark version that includes modifications to support the specific library, then configure the cluster to use this custom version, despite the potential increase in maintenance and support overhead.