Databricks Certified Data Engineer - Professional
Get started today
Ultimate access to all questions.
Comments
Loading comments...
Ultimate access to all questions.
How can you implement a custom partitioner in a Spark job to ensure even distribution of data across partitions when processing a highly skewed dataset?
A
Apply the repartition method with a column that evenly distributes the data, avoiding custom partitioners.
B
Utilize partitionBy with a lambda function that identifies and distributes skewed keys evenly.
C
By overriding the default hash partitioner with a custom partitioner that assigns more partitions to skewed keys.
D
Create a UDF that tags data rows with a partition number, then use repartitionAndSortWithinPartitions.