
Explanation:
The query's performance issue stems from the lack of a time constraint in the join condition, leading to an unbounded state size. Option A correctly introduces a time constraint (clickTime >= impressionTime AND clickTime <= impressionTime + interval 1 hour) that aligns with the watermark strategy, effectively limiting the state size and enabling state cleanup. This is the most efficient solution among the provided options. Options B and D propose time constraints that are either illogical or remove necessary watermarks, which are crucial for state management in streaming applications. Option C suggests an exact time match, which is impractical for streaming data where exact matches are rare and would not significantly reduce the state size.
Ultimate access to all questions.
No comments yet.
A data engineer needs to correlate advertisement impressions with user clicks by joining two streaming DataFrames. The Impressions stream has a watermark set on "event_time" for 10 minutes. The current implementation is:
impressions \
.groupBy(
window("event_time", "5 minutes"),
"id") \
.count() \
.withWatermark("event_time", "2 hours") \
.join(clicks, expr("clickAdId = impressionAdId"), "inner")
impressions \
.groupBy(
window("event_time", "5 minutes"),
"id") \
.count() \
.withWatermark("event_time", "2 hours") \
.join(clicks, expr("clickAdId = impressionAdId"), "inner")
The query performance is degrading significantly. What solution would improve its performance?
A
Joining on event time constraint: clickTime >= impressionTime AND clickTime <= impressionTime interval 1 hour
B
Joining on event time constraint: clickTime + 3 hours < impressionTime - 2 hours
C
Joining on event time constraint: clickTime == impressionTime using a leftOuter join
D
Joining on event time constraint: clickTime >= impressionTime - interval 3 hours and removing watermarks