In complex multi-join queries with varying sizes of intermediate results, what is the most effective method to dynamically adjust shuffle partitions before each join operation to optimize performance?

Real Exam

Adjust spark.sql.adaptive.coalescePartitions.enabled before each join operation based on runtime statistics.

22.3%

Utilize repartition before each join based on anticipated result sizes, informed by profiling previous runs.

56.3%

Pre-calculate the size of join inputs and use spark.sql.shuffle.partitions to set partitions dynamically via a UDF.

17.5%

Maintain a static number of shuffle partitions, relying on Spark's cost optimizer to automatically handle partition sizing.

3.9%

Databricks Certified Data Engineer - Professional