Databricks Certified Data Engineer - Professional
Comments
Loading comments...
Ultimate access to all questions.
You are a data engineer working on optimizing a Spark application to improve its performance on Azure Databricks. The application processes large datasets and has been experiencing slow performance. You decide to use the Spark UI to identify and address performance bottlenecks. Considering the need for a comprehensive analysis that includes data distribution, resource allocation, and task execution, which of the following strategies would you employ to effectively optimize the application? (Choose one)
A
Focus exclusively on the 'Jobs' tab to identify and address only the longest-running stages and tasks, without considering other potential bottlenecks.
B
Use the 'Stages' and 'Tasks' tabs to identify stages with high execution times and skewed task distribution, but overlook the importance of data locality and its impact on performance.
C
Analyze the 'Storage' tab to understand data distribution and partitioning, and the 'Environment' tab to assess resource allocation. Use this information to optimize data partitioning and resource usage, addressing both data skew and resource constraints.
D
Examine the 'Executors' tab to identify executors with low task execution rates, ignoring the potential effects of data skew and resource constraints on overall performance.