Answer-first summary for fast verification
Answer: Create a BigQuery scheduled query to run the SQL transformation with schedule options that repeats every two hours, and enable notification to Pub/Sub topic. Use Pub/Sub and Cloud Functions to send an email after three failed executions.
The correct answer is D because using a BigQuery scheduled query leverages BigQuery's native scheduling capabilities to run the SQL job every two hours. By enabling notification to a Pub/Sub topic and using Pub/Sub and Cloud Functions, you can tailor the retry logic and notification handling, including the specific requirement of sending an email after three failed executions. This approach covers the requirement of handling errors and sending notifications after consecutive failures, which cannot be fully addressed by simply using email_on_failure or retry parameters in Cloud Composer operators.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
No comments yet.
You are tasked with creating an automated SQL data pipeline using BigQuery. The pipeline must perform an aggregate SQL transformation on a specified BigQuery table every two hours and subsequently append the transformation results to another pre-existing BigQuery table. Additionally, the pipeline needs to be robust, capable of retrying in case of errors. It is crucial that the pipeline is also configured to send an email notification after it encounters three consecutive failures. What steps should you take to achieve this configuration?
A
Use the BigQueryUpsertTableOperator in Cloud Composer, set the retry parameter to three, and set the email_on_failure parameter to true.
B
Use the BigQueryInsertJobOperator in Cloud Composer, set the retry parameter to three, and set the email_on_failure parameter to true.
C
Create a BigQuery scheduled query to run the SQL transformation with schedule options that repeats every two hours, and enable email notifications.
D
Create a BigQuery scheduled query to run the SQL transformation with schedule options that repeats every two hours, and enable notification to Pub/Sub topic. Use Pub/Sub and Cloud Functions to send an email after three failed executions.