Google Professional Data Engineer
Get started today
Ultimate access to all questions.
Ultimate access to all questions.
You are responsible for implementing security best practices within your data pipeline system. At present, you manually execute jobs with Project Owner privileges. Your goal is to automate these tasks, which involve handling nightly batch files that contain non-public information stored in Google Cloud Storage. These files need to be processed using a Spark Scala job on a Google Cloud Dataproc cluster, and the resulting output should be deposited into Google BigQuery. What steps should you take to securely and efficiently automate this workload?
Explanation:
The best practice for securely running automated jobs is to use a service account with the minimum necessary permissions to perform the required tasks. In this case, the service account should have the ability to read the batch files from Google Cloud Storage and write to Google BigQuery. This follows the principle of least privilege, ensuring that the service account only has access to the resources it needs to complete the job, thereby enhancing security. Option C is the correct answer as it allows automation with the necessary permissions while adhering to security best practices.