You are tasked with designing a data pipeline responsible for publishing application events to a Google Cloud Pub/Sub topic. While the order of these messages is not a priority, it is crucial to aggregate the events across distinct hourly intervals before the data is eventually loaded into BigQuery for further analysis. Considering the necessity to handle potentially large volumes of events and ensure scalability, which technology should you employ for processing and loading the aggregated data into BigQuery?

Exam-Like

Create a Cloud Function to perform the necessary data processing that executes using the Pub/Sub trigger every time a new message is published to the topic.

9.8%

Schedule a Cloud Function to run hourly, pulling all available messages from the Pub/Sub topic and performing the necessary aggregations.

7.3%

Schedule a batch Dataflow job to run hourly, pulling all available messages from the Pub/Sub topic and performing the necessary aggregations.

26.8%

Create a streaming Dataflow job that reads continually from the Pub/Sub topic and performs the necessary aggregations using tumbling windows.

56.1%

Google Professional Data Engineer

Get started today

Comments