A data engineer discovers that a critical field from a Kafka source was omitted during ingestion into Delta Lake, causing it to be absent in all downstream storage. Although the field existed in the Kafka source, the Kafka service has a retention period of only seven days, while the pipeline has been running for three months.

How can Delta Lake be utilized to prevent this type of data loss in the future?

Real Exam

Last updated: January 6, 2026 at 15:42

Ingesting all raw data and metadata into a bronze Delta table to create a permanent, replayable history of the data state.

Utilizing Delta Lake schema evolution to retroactively compute values for newly added fields from the original source.

Relying on the Delta transaction log and Structured Streaming checkpoints to maintain a complete history of the Kafka producer.

Enabling a setting in Delta Lake that ensures all fields from the source data are automatically included in the ingestion layer.

Databricks Certified Data Engineer - Professional