Ultimate access to all questions.
A data engineer in Databricks needs to schedule a notebook to run daily, passing the current date as a parameter. What is the most appropriate feature to use and how should the parameter be passed?
Explanation:
The correct approach is to use Databricks Jobs and specify the date parameter in the job configuration. This method is native to Databricks and directly supports scheduling notebooks with parameters. Other options either involve external tools (Azure Data Factory, Apache Airflow) or are not suited for automated job scheduling (using %run command with dbutils.widgets.get("date")). Cron jobs, while capable of scheduling, are not a direct feature of Databricks and do not facilitate parameter passing as required.