Answer-first summary for fast verification
Answer: Use CoGroupByKey instead of the SideInput.
The correct answer is D. Using CoGroupByKey instead of SideInput is beneficial for handling large datasets. SideInputs require the entire dataset to be available to each worker, which can lead to performance bottlenecks. CoGroupByKey, on the other hand, groups elements by key and processes each key-group separately, making it more efficient for joining large datasets. It distributes the workload more evenly across Dataflow workers, improving scalability, resource utilization, and overall performance, thereby expediting the Dataflow job.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
You are currently testing a Google Cloud Dataflow pipeline designed to ingest and transform text files. These text files are compressed using gzip, and any errors encountered during the process are written to a dead-letter queue. Additionally, this pipeline makes use of SideInputs to join data. However, you have observed that the pipeline is taking longer to finish than anticipated. What steps should you take to expedite the Dataflow job?
A
Switch to compressed Avro files.
B
Reduce the batch size.
C
Retry records that throw an error.
D
Use CoGroupByKey instead of the SideInput.
No comments yet.