Answer-first summary for fast verification
Answer: Utilize a single job to configure both a DLT Pipeline task and a Notebook task, employing two distinct tasks within the same job with a linear dependency.
It is possible to configure both a DLT Pipeline task and a Notebook task within a single job, arranging them with a linear dependency. This approach allows for the sequential execution of tasks, where the completion of one task triggers the start of the next. Task dependencies can be defined using the 'Depends on' dropdown menu, enabling the creation of a Directed Acyclic Graph (DAG) that outlines the execution order. This method ensures that upstream tasks are completed before downstream tasks begin, with the system optimizing for parallel execution where feasible. Therefore, the optimal solution is to use a single job with two different tasks arranged in a linear dependency.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
A data science team aims to construct a machine learning pipeline where clickstream and orders data will be ingested using Delta Live Tables (DLT). Following this, a Notebook containing Python logic needs to be executed for data transformation, feature extraction, and then training and testing the predictive model. They wish to orchestrate this entire process using Databricks Workflows. Which of the following methods can the team employ to achieve this goal?
A
Establish two separate jobs: run the DLT Pipeline job in triggered mode and, upon its success, manually initiate the Notebook job.
B
Utilize a single job to configure both a DLT Pipeline task and a Notebook task, employing two distinct tasks within the same job with a linear dependency.
C
Since a single job cannot include both a DLT Pipeline task and a Notebook task, use two separate jobs with a linear dependency.
D
Employ a single job to set up both a DLT Pipeline task and a Notebook task, running two different tasks in the same job in parallel.
E
There is no feasible method to orchestrate such a workflow in Databricks.
No comments yet.