You are tasked with creating a Databricks job that involves data extraction from an API, data transformation, and loading the results into a data lake. Each step has dependencies on the previous steps. Describe how you would design this job to ensure efficient and reliable execution.

Simulated

Create a single notebook that includes all steps, execute the notebook as a job, and handle dependencies within the notebook code.

6.0%

Create a multi-task job with separate tasks for each step, configure task dependencies to enforce the sequence, and use shared storage for data exchange between tasks.

86.8%

Schedule each step as a separate job, manually trigger each job based on the completion of the previous step.

4.0%

Combine all steps into a single Python script, upload the script to Databricks, and run it as a job.

3.2%

Databricks Certified Data Engineer - Professional

Get started today

Comments

You are tasked with creating a Databricks job that involves data extraction from an API, data transformation, and loading the results into a data lake. Each step has dependencies on the previous steps. Describe how you would design this job to ensure efficient and reliable execution.