Ultimate access to all questions.
Upgrade Now 🚀
Sign in to unlock AI tutor
What does the default value of 200 for spark.sql.shuffle.partitions signify?
spark.sql.shuffle.partitions
A
By default, all DataFrames in Spark will be spit to perfectly fill the memory of 200 executors.
B
By default, new DataFrames created by Spark will be split to perfectly fill the memory of 200 executors.
C
By default, Spark will only read the first 200 partitions of DataFrames to improve speed.
D
By default, all DataFrames in Spark, including existing DataFrames, will be split into 200 unique segments for parallelization.
E
By default, DataFrames will be split into 200 unique partitions when data is being shuffled.
