To ensure your Spark jobs on Databricks are both cost-effective and scalable while managing varying ETL workloads throughout the day, which strategy would you adopt for dynamically scaling resources based on workload demands?

Real Exam

Utilizing spot instances for Spark clusters whenever possible, with a fallback strategy to on-demand instances for critical jobs.

1.5%

Implementing a custom monitoring solution that adjusts clusters based on CPU and memory utilization, integrating with Azure Cost Management for budget tracking.

1.5%

Leveraging Databricks‘ autoscaling feature to automatically adjust the number of nodes in a cluster based on the workload, with cost constraints defined in the job configuration.

95.4%

Scheduling Spark jobs during off-peak hours using Databricks jobs scheduler to capitalize on lower compute costs and reduced contention.

1.5%

Databricks Certified Data Engineer - Professional

Get started today

Comments

To ensure your Spark jobs on Databricks are both cost-effective and scalable while managing varying ETL workloads throughout the day, which strategy would you adopt for dynamically scaling resources based on workload demands?