You are tasked with creating a robust data pipeline designed to facilitate data sharing between two distinct types of applications: job generators and job runners. The data pipeline must be scalable to handle rising usage demands and should seamlessly integrate new applications without degrading the performance of current applications. What steps should you take to achieve this?