Databricks Certified Data Engineer - Professional
Get started today
Ultimate access to all questions.
A production Structured Streaming job is currently processing each microbatch in under 3 seconds during normal operation, but experiences inconsistent execution times during peak hours—sometimes exceeding 30 seconds per microbatch. The streaming write is configured with a 10-second trigger interval.
Assuming all other variables remain constant and records must be processed in under 10 seconds, which configuration adjustment will satisfy this requirement?
A production Structured Streaming job is currently processing each microbatch in under 3 seconds during normal operation, but experiences inconsistent execution times during peak hours—sometimes exceeding 30 seconds per microbatch. The streaming write is configured with a 10-second trigger interval.
Assuming all other variables remain constant and records must be processed in under 10 seconds, which configuration adjustment will satisfy this requirement?
Explanation:
Option A is correct because decreasing the trigger interval to 5 seconds means batches will be created more frequently, so each batch will have fewer records. This helps ensure that each batch can be processed in less than 10 seconds, even during peak times when processing can slow down. This adjustment directly addresses the requirement to process records within 10 seconds.