Databricks Certified Data Engineer - Associate
Get started today
Ultimate access to all questions.
Deep dive into the quiz with AI chat providers.
We prepare a focused prompt with your quiz and certificate details so each AI can offer a more tailored, in-depth explanation.
Question 38
A data engineer has set up two Jobs that each run nightly. The first Job starts at 12:00 AM, and it usually completes in about 20 minutes. The second Job depends on the first Job, and it starts at 12:30 AM. Sometimes, the second Job fails when the first Job does not complete by 12:30 AM.
Which of the following approaches can the data engineer use to avoid this problem?
Real Exam
Community
LLeetQuiz
A
They can utilize multiple tasks in a single job with a linear dependency
B
They can use cluster pools to help the Jobs run more efficiently
C
They can set up a retry policy on the first Job to help it run more quickly
D
They can limit the size of the output in the second Job so that it will not fail as easily
E
They can set up the data to stream from the first Job to the second Job
Explanation:
Explanation
The correct answer is A because:
-
Multiple tasks in a single job with linear dependency ensures that the second task only starts after the first task successfully completes. This eliminates the timing issue where the second job starts before the first job finishes.
-
Option B (cluster pools) might improve efficiency but doesn't solve the dependency timing problem.
-
Option C (retry policy) might help with transient failures but doesn't guarantee the first job completes before the second job starts.
-
Option D (limit output size) addresses potential performance issues but doesn't solve the fundamental dependency timing problem.
-
Option E (streaming data) is not appropriate for batch jobs and doesn't address the job dependency issue.
By using multiple tasks within a single job with linear dependencies, the data engineer can ensure proper execution order without relying on fixed start times that may cause race conditions.
Comments
Loading comments...