A data scientist is developing a Databricks notebook that requires extensive feature engineering, such as creating new columns and applying transformations. They aim to encapsulate this feature engineering logic into a reusable component. What is the best approach to accomplish this?

Real Exam

Save the intermediate DataFrame and load it in other notebooks as needed.

0.0%

Use the map function to apply the feature engineering transformations.

7.1%

Define a custom PySpark UDF (User-Defined Function) and apply it to the DataFrame.

21.4%

Create a Spark MLlib Transformer class for the feature engineering logic.

71.4%

Databricks Certified Machine Learning - Associate