Answer-first summary for fast verification
Answer: ```option("mergeSchema", True)```
Schema evolution is a feature that enables the addition of newly detected fields to the table. This functionality is activated by including `.option('mergeSchema', 'true')` in your `.write` or `.writeStream` Spark command. For more details, refer to the [Databricks documentation on schema updates](https://docs.databricks.com/delta/update-schema.html#add-columns-with-automatic-schema-update).
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
A data engineer is looking to use Autoloader for ingesting input data into a target table with the ability to automatically evolve the table's schema upon detection of new fields. The following query is provided with a blank to be filled:
.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", checkpointPath)
.load(source_path)
.writeStream
.option("checkpointLocation", checkpointPath)
.___________
.start("target_table")
.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", checkpointPath)
.load(source_path)
.writeStream
.option("checkpointLocation", checkpointPath)
.___________
.start("target_table")
Which option correctly fills in the blank to meet the specified requirement?
A
option("cloudFiles.schemaEvolutionMode", "addNewColumns")
B
option("cloudFiles.mergeSchema", True)
C
option("mergeSchema", True)
D
schema(schema_definition, mergeSchema=True)
E
Autoloader can not automatically evolve the schema of the table when new fields are detected
No comments yet.