
Explanation:
SELECT DISTINCT * FROM table_name will filter out duplicate records from the query execution result set in Spark SQL. DROP DUPLICATES is a DataFrame API function (e.g., df.dropDuplicates()), not a standard SQL command. To permanently remove duplicates, one would typically use a MERGE operation or window functions to filter and overwrite the data.
A
DROP DUPLICATES
B
REMOVE DUPLICATES
C
DELETE FROM table_name WHERE count > 1
D
SELECT DISTINCT * FROM table_name
No comments yet.