
Ultimate access to all questions.
You are tasked with optimizing an existing ETL pipeline that processes large volumes of sensor data from IoT devices. The current pipeline suffers from high latency and processing bottlenecks. Describe the steps you would take to optimize this pipeline, including the tools and techniques you would use to improve its performance and reliability.
A
Increase the batch size of data processed at once without considering other optimizations.
B
Optimize data ingestion with AWS Kinesis, use Apache Spark for parallel processing, and implement auto-scaling for the ETL jobs.
C
Reduce the amount of data processed to decrease latency.
D
Switch to a different cloud provider without changing the pipeline architecture.