Google Professional Data Engineer
Get started today
Ultimate access to all questions.
Deep dive into the quiz with AI chat providers.
We prepare a focused prompt with your quiz and certificate details so each AI can offer a more tailored, in-depth explanation.
NO.42 You want to automate execution of a multi-step data pipeline running on Google Cloud. The pipeline includes Cloud Dataproc and Cloud Dataflow jobs that have multiple dependencies on each other. You want to use managed services where possible, and the pipeline will run every day. Which tool should you use?
Real Exam
Community
LLeetQuiz
A
cron
B
Cloud Composer
C
Cloud Scheduler
D
Workflow Templates on Cloud Dataproc
Explanation:
Explanation
Cloud Composer is the correct choice for this scenario because:
- Managed Service: Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow
- Complex Dependencies: It can handle multi-step pipelines with complex dependencies between different services like Cloud Dataproc and Cloud Dataflow
- Daily Scheduling: Supports scheduled execution (every day) with robust scheduling capabilities
- Service Integration: Has built-in operators for Google Cloud services including Dataproc and Dataflow
- Monitoring and Management: Provides monitoring, logging, and error handling for complex workflows
Why other options are not suitable:
- cron: Basic scheduling but lacks dependency management and monitoring capabilities
- Cloud Scheduler: Good for simple HTTP-based triggers but not designed for complex multi-step workflows with dependencies
- Workflow Templates on Cloud Dataproc: Only manages Dataproc workflows, cannot coordinate between Dataproc and Dataflow jobs
Cloud Composer provides the comprehensive orchestration needed for multi-service data pipelines with complex dependencies.
Comments
Loading comments...