Explanation:
Schema evolution is a feature that enables the addition of newly detected fields to the table. This functionality is activated by including .option('mergeSchema', 'true') in your .write or .writeStream Spark command. For more details, refer to the Databricks documentation on schema updates.
Ultimate access to all questions.
A data engineer is looking to use Autoloader for ingesting input data into a target table with the ability to automatically evolve the table's schema upon detection of new fields. The following query is provided with a blank to be filled:
.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", checkpointPath)
.load(source_path)
.writeStream
.option("checkpointLocation", checkpointPath)
.___________
.start("target_table")
.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", checkpointPath)
.load(source_path)
.writeStream
.option("checkpointLocation", checkpointPath)
.___________
.start("target_table")
Which option correctly fills in the blank to meet the specified requirement?
A
option("cloudFiles.schemaEvolutionMode", "addNewColumns")
B
option("cloudFiles.mergeSchema", True)
C
option("mergeSchema", True)
D
schema(schema_definition, mergeSchema=True)
E
Autoloader can not automatically evolve the schema of the table when new fields are detected
No comments yet.