Ultimate access to all questions.
Upgrade Now 🚀
Sign in to unlock AI tutor
Identify the error in the following code block intended to configure the number of partitions for wide transformations like join() to 32:
join()
Code block:
spark.conf.set("spark.sql.shuffle.partitions", "32")
A
spark.default.parallelism is not the right Spark configuration parameter – spark.sql.shuffle.partitions should be used instead.
B
There is no way to adjust the number of partitions used in wide transformations – it defaults to the number of total CPUs in the cluster.
C
Spark configuration parameters cannot be set in runtime.
D
Spark configuration parameters are not set with spark.conf.set().
E
The second argument should not be the string version of "32" – it should be the integer 32.
