Databricks Certified Data Engineer - Associate
Get started today
Ultimate access to all questions.
A data engineer has set up two Jobs that each run nightly. The first Job starts at 12:00 AM, and it usually completes in about 20 minutes. The second Job depends on the first Job, and it starts at 12:30 AM. Sometimes, the second Job fails when the first Job does not complete by 12:30 AM.
Which of the following approaches can the data engineer use to avoid this problem?
Exam-Like
Community
KKeng
A
They can utilize multiple tasks in a single job with a linear dependency
B
They can use cluster pools to help the Jobs run more efficiently
C
They can set up a retry policy on the first Job to help it run more quickly
D
They can limit the size of the output in the second Job so that it will not fail as easily
E
They can set up the data to stream from the first Job to the second Job
Explanation:
Explanation
The correct answer is A. They can utilize multiple tasks in a single job with a linear dependency.
Why this is correct:
- Task Dependencies: In Databricks Jobs, you can create multiple tasks within a single job and define dependencies between them. By using a linear dependency (Task 2 depends on Task 1), you ensure that Task 2 only starts after Task 1 has completed successfully.
- Eliminates Timing Issues: This approach removes the hard-coded time dependency (12:30 AM start time) and replaces it with a logical dependency based on task completion.
- Built-in Job Orchestration: Databricks Jobs provide native support for task dependencies, making this the most reliable and maintainable solution.
Why other options are incorrect:
- B. Cluster pools: While cluster pools can improve efficiency by reducing cluster startup time, they don't solve the fundamental dependency/timing issue.
- C. Retry policy: A retry policy helps with transient failures but doesn't address the core problem of the second job starting before the first job completes.
- D. Limit output size: This doesn't address the dependency issue; it only potentially reduces the chance of failure due to resource constraints.
- E. Streaming data: Streaming is for real-time data processing, not for batch job dependencies. This would fundamentally change the architecture and isn't appropriate for nightly batch jobs.
Best Practice Recommendation:
In Databricks, when you have jobs with dependencies, it's best to:
- Create a single job with multiple tasks
- Define task dependencies using the
depends_onparameter - Let Databricks handle the orchestration automatically
- Monitor job runs through the Jobs UI for better visibility
Comments
Loading comments...