Explanation:
Adaptive Query Execution (AQE) dynamically optimizes query execution plans based on runtime statistics. One of its features is dynamically coalescing shuffle partitions, which optimizes the number of shuffle partitions based on actual data sizes to help prevent spilling to disk and improve execution speed.
Ultimate access to all questions.
A
Manually set spark.sql.shuffle.partitions to 2000.
B
Increase cluster RAM size.
C
Enable Adaptive Query Execution (AQE) to dynamically coalesce partitions.
D
Convert data to standard Parquet format.
No comments yet.