Databricks Certified Associate Developer for Apache Spark
Get started today
Ultimate access to all questions.
Comments
Loading comments...
Ultimate access to all questions.
Identify the error in the following code block intended to configure the number of partitions for wide transformations like join() to 32:
Code block:
spark.conf.set("spark.sql.shuffle.partitions", "32")
spark.conf.set("spark.sql.shuffle.partitions", "32")
A
spark.default.parallelism is not the right Spark configuration parameter – spark.sql.shuffle.partitions should be used instead.
B
There is no way to adjust the number of partitions used in wide transformations – it defaults to the number of total CPUs in the cluster.
C
Spark configuration parameters cannot be set in runtime.
D
Spark configuration parameters are not set with spark.conf.set().
E
The second argument should not be the string version of "32" – it should be the integer 32.