Databricks Certified Data Engineer - Professional
Get started today
Ultimate access to all questions.
Comments
Loading comments...
Ultimate access to all questions.
In a PySpark application, you are tasked with optimizing the performance of a job that involves performing a series of transformations on a large dataset and subsequently joining it with another dataset. The second dataset is significantly smaller in size. Considering the constraints of cost, compliance, and scalability, which of the following strategies would you choose to ensure the most efficient execution of your job? Choose the best option and explain why it is the most suitable.
A
Implement a broadcast join to combine the large dataset with the small dataset, ensuring the small dataset is broadcasted to all worker nodes to avoid shuffling the large dataset.
B
Opt for a shuffle hash join to join the large dataset with another large dataset, assuming both datasets are of similar size and the join keys are not sorted.
C
Use a sort merge join for joining the large dataset with another large dataset, but only if both datasets are pre-sorted on the join keys to leverage the efficiency of merge operations.
D
Apply a cartesian join to combine the large dataset with any other dataset, disregarding the size of the datasets and the potential performance implications.