In a scenario where you need to manage Spark jobs within an Azure Data Factory pipeline, which of the following best practices should you follow to ensure efficient execution and monitoring of the Spark jobs?