Explanation:
When attempting to rename a column in the upstream 'scorecard' table, the streaming query will immediately stop with an error message: com.databricks.sql.transaction.tahoe.DeltaIllegalStateException: Detected schema change. This occurs even if the query includes the .option('mergeSchema', 'true') setting. The schema change detection prevents the query from continuing, ensuring data integrity is maintained.
Ultimate access to all questions.
No comments yet.
A data engineer executes a streaming query that reads from a Delta table named 'scorecard' and writes to another Delta table named 'performance'. The query includes options for checkpoint location and schema merging. What happens if a column in the 'scorecard' table is renamed while the query is running?
A
The query will continue to run and the column will be renamed in the 'performance' table as well.
B
The query will continue to run but the column name in the 'performance' table will remain the same.
C
The query will be stopped but the column name in the 'performance' table will be updated.
D
The query will be paused for 1 minute to accommodate the changes in the 'performance' table. Once the changes are processed, the query will continue to execute.
E
The query will be stopped and the column name in the 'performance' table will remain the same.