Answer-first summary for fast verification
Answer: Within a single job, add both notebooks as tasks with linear dependency, specifying different clusters for each task
The optimal approach involves creating a single job with both notebooks as tasks, each configured with its own compute cluster to meet specific requirements, and establishing a linear dependency to ensure the second notebook runs only after the first completes successfully. This method provides: 1. **Task-Based Flexibility**: Each notebook operates as an independent task with a tailored compute cluster. 2. **Linear Dependency**: Ensures the correct execution order, preventing errors. 3. **Cluster Configuration**: Allows for resource optimization and cost-effectiveness for each task. 4. **Scheduling and Monitoring**: The job scheduler manages task execution according to the schedule, with tools available for monitoring progress. This strategy efficiently manages notebook execution with differing compute needs, ensuring optimal performance and resource utilization.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
No comments yet.
You are tasked with scheduling two notebooks where the second notebook depends on the first, and each requires different compute resources for optimal performance. What is the most efficient way to configure these notebooks as jobs?
A
Use DELTA LIVE PIPELINES instead of notebook tasks
B
Configure each notebook as a separate job with a single cluster and use job dependencies to link them
C
Within a single job, add both notebooks as tasks with linear dependency, specifying different clusters for each task
D
Set up both notebooks in a single job as individual tasks and use the cluster API to configure the second cluster before the second task starts
E
Utilize a very large cluster within a single job to run both tasks