In the context of optimizing Spark data processing within Databricks for a Microsoft Azure environment, which factor plays a pivotal role in determining the ideal number of partitions for a DataFrame? | Databricks Certified Data Engineer - Professional Quiz - LeetQuiz