Databricks Certified Data Engineer - Professional
Get started today
Ultimate access to all questions.
Comments
Loading comments...
Ultimate access to all questions.
In a scenario where you are tasked with updating multiple records in a Spark table using a Type 1 strategy (overwrite), and you need to consider factors such as performance, data integrity, and minimal downtime, which of the following strategies would be the MOST efficient and correct approach? Choose one option from the four provided.
A
Use the DataFrame.na.fill() method to fill missing values in the updated records and then perform an overwrite operation on the existing table. This method is straightforward but may not efficiently handle updates where records do not have missing values.
B
Use the DataFrame.union() method to combine the updated records with the existing table, ensuring no duplicate records are present, and then overwrite the existing table. This method is efficient for bulk updates where updated records do not overlap with existing ones.
C
Use the DataFrame.join() method to join the updated records with the existing table on a key column and then overwrite the existing table. This method is suitable when you need to update specific records based on a key but may introduce complexity if schemas differ.
D
Use the DataFrame.withColumn() method to add a new column to the updated records indicating the update status and then overwrite the existing table. This method is useful for tracking updates but does not directly facilitate the update process.