Databricks Certified Data Engineer - Professional
Get started today
Ultimate access to all questions.
A data engineer is working on a streaming query that processes orders data. The query is missing a crucial part for handling late-arriving data, specifically to maintain the streaming state information for 30 minutes. The query snippet is as follows:
spark.readStream
.table("orders_cleaned")
____________________________
.groupBy(
"order_timestamp",
"author")
.agg(
count("order_id").alias("orders_count"),
avg("quantity").alias("avg_quantity"))
.writeStream
.option("checkpointLocation", "dbfs:/path/checkpoint")
.table("orders_stats")
Which option correctly fills in the blank to meet the requirement of handling late-arriving data by maintaining the streaming state information for 30 minutes?
A data engineer is working on a streaming query that processes orders data. The query is missing a crucial part for handling late-arriving data, specifically to maintain the streaming state information for 30 minutes. The query snippet is as follows:
spark.readStream
.table("orders_cleaned")
____________________________
.groupBy(
"order_timestamp",
"author")
.agg(
count("order_id").alias("orders_count"),
avg("quantity").alias("avg_quantity"))
.writeStream
.option("checkpointLocation", "dbfs:/path/checkpoint")
.table("orders_stats")
Which option correctly fills in the blank to meet the requirement of handling late-arriving data by maintaining the streaming state information for 30 minutes?
Explanation:
The pyspark.sql.DataFrame.withWatermark function is designed to track state information for a specified window of time, accommodating delays in record arrivals. This makes it the correct choice for handling late-arriving data by maintaining the streaming state information for 30 minutes. For more details, refer to the Apache Spark documentation.