Answer-first summary for fast verification
Answer: The table was set up as an external table.
In Spark SQL, dropping a managed table results in the deletion of both the table's metadata and its data files. Conversely, dropping an external table only removes the metadata, leaving the data files intact. This design prevents accidental loss of data that may be shared across different tables or applications. The persistence of data files after the `DROP TABLE` command indicates that `my_table` was an external table, hence only the metadata was deleted. Managed tables, which are fully under Spark's control, would have both metadata and data files deleted upon dropping the table.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
A data engineer attempts to delete a Spark SQL table named my_table to remove both its metadata and data. After executing the command DROP TABLE IF EXISTS my_table;, the table disappears from the list of tables, but the data files remain. What explains the persistence of the data files despite the metadata deletion?
A
The table's data was under 10 GB.
B
The table was a managed table.
C
The table was specified without a location.
D
The table was set up as an external table.
E
The table's data exceeded 10 GB.
No comments yet.