Analysis of the Question Requirements
The question requires configuring autoscaling for all-purpose clusters in Azure Databricks with specific requirements:
- Scale down after 3 minutes of underutilization
- Minimize scale-up time to maximum workers
- Minimize costs
Evaluation of Options
Option A: Enable container services for workspace1
- Not suitable: Container services are used for custom container images and Docker integration, which is unrelated to autoscaling functionality.
- This option doesn't address any of the autoscaling requirements.
Option B: Upgrade workspace1 to the Premium pricing tier
- Optimal choice: This is the prerequisite for enabling optimized autoscaling, which is required to meet the specific requirements.
- Why it's necessary:
- Standard pricing tier only supports standard autoscaling, which scales down only after 10+ minutes of complete idleness
- Premium tier provides optimized autoscaling that can scale down after 150 seconds (2.5 minutes) of underutilization
- Optimized autoscaling provides faster scale-up to maximum workers
- While Premium tier costs more, it enables cost optimization through proper autoscaling behavior
Option C: Set Cluster Mode to High Concurrency
- Not the first action: High Concurrency mode enables multiple users to share clusters efficiently, but it doesn't enable the required autoscaling behavior.
- This can be configured after enabling the proper autoscaling capabilities.
Option D: Create a cluster policy in workspace1
- Not immediately possible: Cluster policies require Premium tier access. Attempting this first would fail since the workspace is in Standard tier.
- Cluster policies are the mechanism to enforce autoscaling rules, but the foundational capability must be available first.
Why Option B is the Correct First Step
- Prerequisite requirement: Optimized autoscaling (needed for 3-minute scale-down) is only available in Premium tier
- Cost optimization: While Premium tier has higher base cost, it enables significant savings through proper autoscaling behavior that matches workload patterns
- Sequential dependency: Without Premium tier, neither cluster policies nor the required autoscaling behavior can be implemented
- Meets all requirements: Enables fast scale-up, appropriate scale-down timing, and overall cost minimization through efficient resource utilization
The upgrade to Premium tier is the foundational step that unlocks the autoscaling capabilities needed to implement the specific requirements effectively.
