Databricks Certified Data Engineer - Associate

Explanation

In Apache Spark Structured Streaming, the trigger() method is used to specify how often the streaming query should process data. There are several trigger types:

1. ProcessingTime trigger: Processes data at fixed intervals (micro-batch processing)
2. Once trigger: Processes all available data once and then stops
3. Continuous trigger: Processes data continuously (experimental)

For processing data every 5 seconds using micro-batch processing, the correct syntax is:

.trigger(processingTime="5 seconds")

.trigger(processingTime="5 seconds")

Why other options are incorrect:

• A. trigger("5 seconds"): This syntax is not valid in Spark Structured Streaming. The trigger method requires specifying the trigger type.
• B. trigger(): Without parameters, this would use the default trigger which processes data as soon as possible, not at fixed intervals.
• C. trigger(once="5 seconds"): This is incorrect syntax. The once trigger doesn't take a time parameter - it processes all data once and stops.
• E. trigger(continuous="5 seconds"): Continuous trigger is for continuous processing mode, not micro-batch processing. Also, continuous trigger doesn't use time intervals in this way.

Correct usage in the code:

(spark.table("sales")
  .withColumn("avg_price", col("sales") / col("units"))
  .writeStream
  .option("checkpointLocation", checkpointPath)
  .trigger(processingTime="5 seconds")
  .outputMode("complete")
  .table("new_sales"))

(spark.table("sales")
  .withColumn("avg_price", col("sales") / col("units"))
  .writeStream
  .option("checkpointLocation", checkpointPath)
  .trigger(processingTime="5 seconds")
  .outputMode("complete")
  .table("new_sales"))

This configuration will execute a micro-batch every 5 seconds, processing any new data that has arrived since the last micro-batch.