A Spark job is running slower than anticipated. While analyzing the Spark UI, a data engineer observes that for tasks in a specific stage, the Min and Median Durations are approximately equal, but the Max Duration is about 100 times longer than the minimum.

What issue is causing the overall job to take longer to complete?

Powered ByGPT-5

Databricks Certified Data Engineer - Professional