You are tasked with designing a scalable data pipeline to facilitate data exchange between job generators and job runners. The solution must be adaptable to include new applications without affecting the performance of existing ones. Which approach would best meet these requirements?

Real Exam

Explanation:

The correct answer is B: Implement a Cloud Pub/Sub topic for job publishing and subscriptions for job execution. Cloud Pub/Sub is designed for asynchronous messaging between applications, making it an ideal choice for scalable and flexible data pipelines. It allows for the decoupling of job generators and runners, facilitating easy scalability and the integration of new applications without impacting existing ones.

Why the other options are less suitable:

A: Develop an API using App Engine - While App Engine can host APIs, it's not optimized for the asynchronous messaging needs of a data pipeline.
C: Set up a Cloud SQL table - Cloud SQL is a relational database service, not designed for the high scalability and flexibility required by data pipelines.
D: Utilize a Cloud Spanner table - Although Cloud Spanner offers scalability and strong consistency, it's an overcomplicated and costly solution for simple data pipeline needs.

Powered ByGPT-5

Google Professional Data Engineer

Get started today

Google Professional Data Engineer

Get started today

You are tasked with designing a scalable data pipeline to facilitate data exchange between job generators and job runners. The solution must be adaptable to include new applications without affecting the performance of existing ones. Which approach would best meet these requirements?