
Ultimate access to all questions.
An aerospace company utilizes a proprietary data format for storing its flight data. As a data engineer, your task is to establish a connection between this new data source and Google BigQuery. Subsequently, you need to stream the flight data into BigQuery. The objective is to efficiently import the data while minimizing resource consumption. What approach should you take?
A
Write a shell script that triggers a Cloud Function that performs periodic ETL batch jobs on the new data source.
B
Use a standard Dataflow pipeline to store the raw data in BigQuery, and then transform the format later when the data is used.
C
Use Apache Hive to write a Dataproc job that streams the data into BigQuery in CSV format.
D
Use an Apache Beam custom connector to write a Dataflow pipeline that streams the data into BigQuery in Avro format.