You are designing a data pipeline that involves the transformation of large datasets using Spark in Azure Databricks. The pipeline needs to be optimized for performance and cost. Describe the strategies you would employ to manage Spark jobs within the pipeline, including considerations for job scheduling, resource allocation, and cost management.