When optimizing Apache Spark jobs for large-scale data processing in Databricks, you face performance bottlenecks due to data shuffling. Which strategy would you use to reduce data shuffle and enhance job performance?

Real Exam

Configuring the Spark session to use more efficient serialization formats like Parquet for intermediate data storage

7.6%

Implementing a custom partitioner to ensure data is evenly distributed across partitions before shuffling

25.0%

Utilizing broadcast variables to minimize data transfer across nodes during join operations

56.5%

Increasing the size of the Spark executors and the number of cores to speed up the data processing

10.9%

Databricks Certified Data Engineer - Professional