Databricks Certified Data Engineer - Associate
Get started today
Ultimate access to all questions.
A data engineer needs to apply custom logic to identify employees with more than 5 years of experience in array column employees in table stores. The custom logic should create a new column exp_employees that is an array of all of the employees with more than 5 years of experience for each row. In order to apply this custom logic at scale, the data engineer wants to use the FILTER higher-order function.
Which of the following code blocks successfully completes this task?
Real Exam
Community
KKeng
A
SELECT
store_id,
employees,
FILTER (employees, i -> i.years_exp > 5) AS exp_employees
FROM stores;
SELECT
store_id,
employees,
FILTER (employees, i -> i.years_exp > 5) AS exp_employees
FROM stores;
B
SELECT
store_id,
employees,
FILTER (exp_employees, years_exp > 5) AS exp_employees
FROM stores;
SELECT
store_id,
employees,
FILTER (exp_employees, years_exp > 5) AS exp_employees
FROM stores;
C
(Incomplete — no code provided)
Explanation:
Explanation
Option A is correct because it properly uses the FILTER higher-order function with the correct syntax:
- FILTER function syntax:
FILTER(array, lambda_function)where the lambda function takes each element of the array and returns a boolean. - Correct lambda expression:
i -> i.years_exp > 5- This correctly references the array elementiand accesses itsyears_expfield. - Logical flow: It filters the
employeesarray to create a new arrayexp_employeescontaining only employees with more than 5 years of experience.
Option B is incorrect because:
- It tries to filter
exp_employeeswhich doesn't exist yet (it's being created in the same expression). - The lambda function syntax is incorrect - it doesn't properly reference the array element.
- It would result in a circular reference error.
Key concepts:
- FILTER is a higher-order function in Spark SQL that operates on arrays.
- The lambda function parameter (here
i) represents each element in the array. - You need to access fields of the array elements using dot notation (
i.years_exp). - The result is a new array containing only elements that satisfy the condition.
Comments
Loading comments...