Explanation:
The optimal approach involves creating a single job with both notebooks as tasks, each configured with its own compute cluster to meet specific requirements, and establishing a linear dependency to ensure the second notebook runs only after the first completes successfully. This method provides:
This strategy efficiently manages notebook execution with differing compute needs, ensuring optimal performance and resource utilization.
Ultimate access to all questions.
You are tasked with scheduling two notebooks where the second notebook depends on the first, and each requires different compute resources for optimal performance. What is the most efficient way to configure these notebooks as jobs?
A
Use DELTA LIVE PIPELINES instead of notebook tasks
B
Configure each notebook as a separate job with a single cluster and use job dependencies to link them
C
Within a single job, add both notebooks as tasks with linear dependency, specifying different clusters for each task
D
Set up both notebooks in a single job as individual tasks and use the cluster API to configure the second cluster before the second task starts
E
Utilize a very large cluster within a single job to run both tasks
No comments yet.