How can you optimize the Continuous Integration (CI) pipeline for a large-scale data processing application in Azure Databricks, involving hundreds of notebooks and complex dependencies, to reduce build times while ensuring comprehensive testing coverage?
