Google Professional Data Engineer
Get started today
Ultimate access to all questions.
Ultimate access to all questions.
As a data engineer tasked with designing storage for a data pipeline on Google Cloud, you must handle very large text files. Your design needs to support ANSI SQL queries and enable efficient compression. Furthermore, it is crucial to support parallel loading from the input locations while adhering to Google's recommended best practices. What steps should you take to achieve this?
Explanation:
The correct answer is B. Transforming text files to compressed Avro using Cloud Dataflow and using Cloud Storage with BigQuery permanent linked tables meets all the criteria of supporting ANSI SQL queries, compression, and parallel load. This approach allows efficient storage and querying, taking advantage of Avro's compression capabilities and parallel processing capabilities supported by Cloud Dataflow. Additionally, using Cloud Storage with BigQuery linked tables avoids the additional storage costs and import overhead associated with directly loading large amounts of data into BigQuery.