
Answer-first summary for fast verification
Answer: spark.sql.adaptive.skewedJoin.enabled
The correct answer is A. The property `spark.sql.adaptive.skewedJoin.enabled` is specifically designed to configure whether skewed partitions are automatically detected and subdivided into smaller partitions when joining two DataFrames together. This is part of Spark's Adaptive Query Execution (AQE) feature, which aims to optimize query execution plans at runtime. Option B, `spark.sql.adaptive.coalescePartitions.enable`, is related to coalescing small partitions into larger ones to reduce overhead, not handling skew. Option C, `spark.sql.adaptive.skewHints.enabled`, and option E, `spark.sql.shuffle.skewHints.enabled`, are about enabling skew hints, which are not the same as automatic skew detection and handling. Option D, `spark.sql.shuffle.partitions`, sets the default number of partitions for shuffles, unrelated to skew detection.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
No comments yet.
Which Spark property configures automatic detection and subdivision of skewed partitions into smaller partitions during DataFrame joins?
A
spark.sql.adaptive.skewedJoin.enabled
B
spark.sql.adaptive.coalescePartitions.enable
C
spark.sql.adaptive.skewHints.enabled
D
spark.sql.shuffle.partitions
E
spark.sql.shuffle.skewHints.enabled