Databricks Certified Data Engineer - Professional
Comments
Loading comments...
Ultimate access to all questions.
In the context of monitoring a Spark application to optimize performance, you suspect data skew is affecting your application's efficiency. Data skew can lead to longer processing times for some tasks, impacting overall job performance. Considering the need for a cost-effective solution that ensures compliance with data processing standards and scalability for large datasets, how would you use the Spark UI to identify and diagnose data skew issues within your application? Choose the best option from the following:
A
Navigate to the 'Storage' tab to analyze the distribution of data across tasks, as this tab directly shows data skew by displaying uneven data distribution.
B
Inspect the 'Jobs' tab to identify stages with a significant variance in task durations, which is a strong indicator of data skew, and then consider repartitioning the data to balance the load.
C
Use the 'Environment' tab to review the configuration settings, as incorrect settings can mimic the symptoms of data skew, and adjust them accordingly.
D
Examine the 'Executors' tab to spot executors handling disproportionately large amounts of data, which suggests data skew, and then increase the number of executors to distribute the load more evenly.