In a data engineering project, you need to implement a solution for managing and deploying data pipelines in a scalable and efficient manner. Which of the following design patterns would be most suitable for this task, and why?