Databricks Certified Data Engineer - Associate

Explanation

In Apache Spark Structured Streaming, to perform a streaming read (as opposed to a batch read), you need to use .readStream instead of .read.

Key Points:

• .read is used for batch processing
• .readStream is used for streaming processing
• The .format("cloudFiles") is correct for reading from cloud storage with Auto Loader
• The .option("cloudFiles.format", "json") is correct for specifying JSON format
• The .schema(schema) and .load(dataSource) are properly configured

Why other options are incorrect:

• Option B: There is no .stream method in Spark's DataFrameReader API
• Option C: .format("stream") is not a valid format - "cloudFiles" is the correct format for Auto Loader
• Option D: There is no .stream method that can be added after spark
• Option E: Adding .stream after .load(dataSource) would be syntactically incorrect

The corrected code should be:

(spark
 .readStream
 .schema(schema)
 .format("cloudFiles")
 .option("cloudFiles.format", "json")
 .load(dataSource)
)

(spark
 .readStream
 .schema(schema)
 .format("cloudFiles")
 .option("cloudFiles.format", "json")
 .load(dataSource)
)