
Explanation:
The correct answer is A. The property spark.sql.adaptive.skewedJoin.enabled is specifically designed to configure whether skewed partitions are automatically detected and subdivided into smaller partitions when joining two DataFrames together. This is part of Spark's Adaptive Query Execution (AQE) feature, which aims to optimize query execution plans at runtime. Option B, spark.sql.adaptive.coalescePartitions.enable, is related to coalescing small partitions into larger ones to reduce overhead, not handling skew. Option C, spark.sql.adaptive.skewHints.enabled, and option E, spark.sql.shuffle.skewHints.enabled, are about enabling skew hints, which are not the same as automatic skew detection and handling. Option D, spark.sql.shuffle.partitions, sets the default number of partitions for shuffles, unrelated to skew detection.
Ultimate access to all questions.
Which Spark property configures automatic detection and subdivision of skewed partitions into smaller partitions during DataFrame joins?
A
spark.sql.adaptive.skewedJoin.enabled
B
spark.sql.adaptive.coalescePartitions.enable
C
spark.sql.adaptive.skewHints.enabled
D
spark.sql.shuffle.partitions
E
spark.sql.shuffle.skewHints.enabled
No comments yet.