Databricks Certified Data Engineer - Associate
Get started today
Ultimate access to all questions.
You are working with a Delta Lake table 'transactions' that contains duplicate rows. Write a PySpark code snippet to deduplicate the rows based on all columns and save the result back to the same table.
You are working with a Delta Lake table 'transactions' that contains duplicate rows. Write a PySpark code snippet to deduplicate the rows based on all columns and save the result back to the same table.
Simulated
A
df = spark.read.format('delta').load('/path/to/transactions') df = df.dropDuplicates() df.write.format('delta').mode('overwrite').save('/path/to/transactions')
61.8%
B
df = spark.read.format('delta').load('/path/to/transactions') df = df.distinct() df.write.format('delta').mode('overwrite').save('/path/to/transactions')
Comments
Loading comments...