Databricks Certified Data Engineer - Associate
Get started today
Ultimate access to all questions.
A data engineer has configured a Structured Streaming job to read from a table, manipulate the data, and then perform a streaming write into a new table.
The code block used by the data engineer is below:
(spark.table("sales")
.withColumn("avg_price", col("sales") / col("units"))
.writeStream
.option("checkpointLocation", checkpointPath)
.outputMode("complete")
.______
.table("new_sales"))
(spark.table("sales")
.withColumn("avg_price", col("sales") / col("units"))
.writeStream
.option("checkpointLocation", checkpointPath)
.outputMode("complete")
.______
.table("new_sales"))
If the data engineer only wants the query to execute a single micro-batch to process all of the available data, which of the following lines of code should the data engineer use to fill in the blank?
Exam-Like
Community
KKeng
A
trigger(once=True)
B
trigger(continuous="once")
C
processingTime("once")
D
trigger(processingTime="once")
E
processingTime(1)
Explanation:
Explanation
In Apache Spark Structured Streaming, when you want to execute a streaming query that processes all available data only once (a single micro-batch), you should use the trigger(once=True) option.
Key Points:
-
trigger(once=True)- This is the correct syntax for Scala/Java API to trigger a single micro-batch that processes all available data and then stops the query. -
Other options explained:
- Option B (
trigger(continuous="once")): Incorrect syntax. Continuous processing mode doesn't use "once" parameter. - Option C (
processingTime("once")): Incorrect.processingTimeexpects a time interval string like "1 second", "5 minutes", etc., not "once". - Option D (
trigger(processingTime="once")): Incorrect syntax.processingTimeshould be a method, not a parameter. - Option E (
processingTime(1)): This would trigger micro-batches every 1 second continuously, not just once.
- Option B (
-
How it works:
- When
trigger(once=True)is used, the streaming query:- Processes all available data in the source
- Executes exactly one micro-batch
- Then automatically stops the query
- This is useful for batch-like processing using the streaming API
- When
-
Note on Scala syntax: In the provided Scala code, the correct syntax would be
.trigger(Trigger.Once())or.trigger(once=True)depending on the exact Spark version and API usage. The optionAwithtrigger(once=True)is the conceptually correct answer among the given choices.
This trigger mode is particularly useful for:
- Processing all available data at once
- Running periodic batch jobs using streaming infrastructure
- Testing and development scenarios
Comments
Loading comments...