Answer-first summary for fast verification
Answer: Implement a retry policy specifically for the task that fails periodically
In this scenario, the job fails due to a specific task. Implementing a retry policy for that task allows for a specified number of retry attempts before the job is considered failed. This approach targets the root cause without unnecessarily increasing compute costs by retrying the entire job or running multiple instances. The retry policy can be configured under Advanced options by selecting Edit Retry Policy, where the retry interval is calculated in milliseconds between the start of the failed run and the subsequent retry run.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
A data engineering team has an ETL job that runs every midnight but fails intermittently due to a specific task, requiring manual reruns in the morning. This issue is causing significant overhead. What approach can the team take to ensure the job completes every night while minimizing compute costs?
A
Monitor the task during execution to identify the cause of failure
B
Implement a retry policy specifically for the task that fails periodically
C
Apply a retry policy to the entire job
D
Schedule the job to run multiple times to guarantee at least one completion
E
Use a Jobs cluster for each task within the job
No comments yet.