Explanation:
When attempting to rename a column in the upstream 'scorecard' table, the streaming query will immediately stop with an error message: com.databricks.sql.transaction.tahoe.DeltaIllegalStateException: Detected schema change. This occurs even if the query includes the .option('mergeSchema', 'true') setting. The schema change detection prevents the query from continuing, ensuring data integrity is maintained.
A data engineer executes a streaming query that reads from a Delta table named 'scorecard' and writes to another Delta table named 'performance'. The query includes options for checkpoint location and schema merging. What happens if a column in the 'scorecard' table is renamed while the query is running?
A
The query will continue to run and the column will be renamed in the 'performance' table as well.
B
The query will continue to run but the column name in the 'performance' table will remain the same.
C
The query will be stopped but the column name in the 'performance' table will be updated.
D
The query will be paused for 1 minute to accommodate the changes in the 'performance' table. Once the changes are processed, the query will continue to execute.
E
The query will be stopped and the column name in the 'performance' table will remain the same.
No comments yet.