Google Professional Data Engineer
Get started today
Ultimate access to all questions.
Ultimate access to all questions.
You have multiple Spark jobs scheduled to run on a Cloud Dataproc cluster, with some jobs running in sequence and others concurrently. What is the best method to automate this process?
Explanation:
The optimal solution is to use a Cloud Dataproc Workflow Template, which allows for the automation of Spark job sequences and concurrent executions. This approach is complemented by integrating with Apache Airflow DAGs via Cloud Composer for scheduled triggers. Reference: Google Cloud Dataproc Workflow Templates.