Answer-first summary for fast verification
Answer: Estimators fit models on data, while Transformers apply the fitted models to transform data.
Estimators are algorithms that can be fit on a dataset to produce a model, such as a logistic regression. Transformers are algorithms that take a DataFrame and return a new DataFrame with one or more columns added, such as a TF-IDF transformer. The correct use of these components ensures that data is correctly transformed and models are appropriately trained within a pipeline.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
No comments yet.
In the context of training a machine learning model using Spark ML, explain the difference between an Estimator and a Transformer. Provide examples of each and discuss how they interact within a Spark ML pipeline. Additionally, describe a scenario where the correct use of Estimators and Transformers is crucial for the success of the model.
A
Estimators are used for data transformation, while Transformers are used for model fitting.
B
Estimators and Transformers are interchangeable terms in Spark ML.
C
Estimators fit models on data, while Transformers apply the fitted models to transform data.
D
Both Estimators and Transformers are used only for data visualization.