Using an all-purpose cluster is acceptable during development, but for scheduled jobs where interaction with the notebook isn't necessary, switching to a job cluster can significantly reduce costs. The difference in cost arises from the DBU (Databricks Unit) charges, not the compute costs from the cloud provider. Job clusters are less expensive in terms of DBU costs compared to all-purpose clusters. For example, AWS charges $0.15 per DBU for job clusters versus$0.55 per DBU for all-purpose clusters. To check the DBU cost for your cluster, look at the cluster details in the top right corner.