Databricks Certified Data Engineer - Associate
Get started today
Ultimate access to all questions.
Ultimate access to all questions.
A data analyst has crafted a query designed to execute against a Delta table. To ensure the quality and cleanliness of the data returned by this query, they seek assistance from the data engineering team to establish a series of validation tests. Notably, the data engineering team conducts its testing using Python instead of SQL. Which of the following operations would allow the data engineering team to execute the query and manage the resulting data within a PySpark environment?
Explanation:
The correct operation to run the query and operate with the results in PySpark is 'spark.sql'. This function allows you to execute SQL queries from within PySpark. While 'spark.table' can load a table and 'SELECT * FROM sales' is a SQL query, neither option allows executing raw SQL from a PySpark context. Additionally, using 'spark.delta.table' is not correct because this function does not exist, and saying there is no way to share data between PySpark and SQL is incorrect as well.