Databricks Certified Data Engineer - Professional
Get started today
Ultimate access to all questions.
Comments
Loading comments...
Ultimate access to all questions.
A senior data engineer is troubleshooting performance delays in a Structured Streaming pipeline that ingests data from Apache Kafka into a Delta Lake table. To improve observability, the engineer updates the ingestion logic and the Delta table schema to include the ingestion timestamp, the Kafka topic name, and the partition ID.
What specific limitation will the team face when implementing this change on the existing table?
A
The Delta transaction log metadata will be invalidated, necessitating a manual recovery of the table state.
B
A non-null default value must be explicitly provided for every new column during the schema evolution process.
C
Spark's Kafka source connector is natively unable to extract metadata fields like topic and partition into DataFrame columns.
D
The newly added columns will not be backfilled; historical records will contain NULL for these fields.
E
Delta Lake does not support adding new columns to existing production tables without a full table overwrite.