Explanation:
The correct approach for enforcing schema consistency in a streaming application writing to Delta Lake without disrupting the stream is to utilize the mergeSchema option on each write operation. This option automatically updates the table's schema to include new columns from the incoming data, ensuring schema evolution is handled in real-time without manual intervention. This method is efficient, maintains data integrity, and avoids the need for maintenance windows or batch processing for schema validation.
Ultimate access to all questions.
What is the best approach to enforce schema consistency in a streaming application that writes to Delta Lake, ensuring schema evolution does not disrupt the stream?
A
Stream data to a raw Delta table and use batch jobs for schema validation and data movement to a production table.
B
Implement a schema registry that dynamically updates the Delta table schema before data ingestion.
C
Utilize mergeSchema option on each write operation, allowing automatic schema updates.
D
Disable schema enforcement, relying on manual schema updates during maintenance windows.
No comments yet.