Explanation:
AWS Glue Python shell jobs run on a single node and are designed for lightweight ETL workloads with small datasets (under ~100 MB per file is ideal). They are significantly cheaper than Spark-based Glue jobs (which provision a distributed cluster) or EMR clusters, and avoid the operational overhead of managing EKS or EMR. For daily small CSV files, pandas in a Python shell job is the most cost-effective choice.
Ultimate access to all questions.
Question 44.
A data engineer needs to build an extract, transform, and load (ETL) job. The ETL job will process daily incoming .csv files that users upload to an Amazon S3 bucket. The size of each S3 object is less than 100 MB. Which solution will meet these requirements MOST cost-effectively?
A
Write a custom Python application. Host the application on an Amazon Elastic Kubernetes Service (Amazon EKS) cluster.
B
Write a PySpark ETL script. Host the script on an Amazon EMR cluster.
C
Write an AWS Glue PySpark job. Use Apache Spark to transform the data.
D
Write an AWS Glue Python shell job. Use pandas to transform the data.
No comments yet.