Databricks Certified Data Engineer - Professional

Get started today

Ultimate access to all questions.

Explanation:

The correct answer is C. Increasing spark.sql.shuffle.partitions and enabling spark.shuffle.compress. This combination is optimal because:

Increasing spark.sql.shuffle.partitions helps distribute data more evenly across partitions, reducing the amount of data shuffled in each partition and thereby minimizing shuffle spill and network I/O.
Enabling spark.shuffle.compress reduces the volume of data transferred over the network during shuffle operations by compressing the data beforehand, which enhances performance.

While options A, B, and D involve important Spark settings for general optimization, they are not as directly effective for the specific goal of reducing shuffle spill and network I/O in jobs with extensive shuffle operations as option C.

Explanation:

The correct answer is C. Increasing spark.sql.shuffle.partitions and enabling spark.shuffle.compress. This combination is optimal because:

Increasing spark.sql.shuffle.partitions helps distribute data more evenly across partitions, reducing the amount of data shuffled in each partition and thereby minimizing shuffle spill and network I/O.
Enabling spark.shuffle.compress reduces the volume of data transferred over the network during shuffle operations by compressing the data beforehand, which enhances performance.

Comments (0)

No comments yet.

In a Spark job that involves extensive shuffle operations due to wide transformations, which combination of settings is most effective for reducing shuffle spill and network I/O?

Real Exam

Tuning spark.executor.memory and spark.shuffle.file.buffer

16.1%

Adjusting spark.memory.fraction and spark.reducer.maxSizeInFlight

11.9%

Increasing spark.sql.shuffle.partitions and enabling spark.shuffle.compress

58.5%

Configuring spark.default.parallelism and spark.shuffle.spill.compress

13.6%