Databricks Certified Data Engineer - Associate
Get started today
Ultimate access to all questions.
A data engineer has a Python variable table_name that they would like to use in a SQL query. They want to construct a Python code block that will run the query using table_name. They have the following incomplete code block: ____(F"SELECT customer_id, spend FROM (table_name)")
Which of the following can be used to fill in the blank to successfully complete the task?
Real Exam
Community
KKeng
A
spark.sql
B
spark.read
C
spark.execute
D
spark.query
E
spark.table
F
spark.run
Explanation:
Explanation
The correct answer is A. spark.sql.
Why spark.sql is correct:
- spark.sql() is the primary method in PySpark for executing SQL queries on DataFrames and tables.
- The query string uses an f-string (F"SELECT ...") which allows Python variable interpolation, so
(table_name)will be replaced with the actual value of the Python variabletable_name. - The syntax
spark.sql("SELECT ... FROM table_name")is the standard way to run SQL queries in PySpark.
Why other options are incorrect:
- B. spark.read: This is used for reading data from various sources (CSV, Parquet, etc.), not for executing SQL queries.
- C. spark.execute: This method doesn't exist in PySpark's SparkSession API.
- D. spark.query: This method doesn't exist in PySpark's SparkSession API.
- E. spark.table: This method returns a DataFrame for a given table name but doesn't execute arbitrary SQL queries.
- F. spark.run: This method doesn't exist in PySpark's SparkSession API.
Complete code example:
# Assuming table_name is a Python variable
table_name = "sales_data"
result = spark.sql(F"SELECT customer_id, spend FROM {table_name}")
# Assuming table_name is a Python variable
table_name = "sales_data"
result = spark.sql(F"SELECT customer_id, spend FROM {table_name}")
Note: The original code shows (table_name) in parentheses, but in an f-string, it should be {table_name}. The correct f-string syntax would be:
spark.sql(F"SELECT customer_id, spend FROM {table_name}")
spark.sql(F"SELECT customer_id, spend FROM {table_name}")
Comments
Loading comments...