The correct answer is D. Word Embeddings. Here's why:
- A. Tokenization: Splits text into tokens (words, phrases) but doesn't convert them into numerical vectors.
- B. Text Encoding: Converts text characters into numerical codes (e.g., ASCII) without capturing semantic meaning.
- C. Feature Extraction: A broad term for techniques to extract features from data, not specifically for text-to-vector conversion.
- D. Word Embeddings (CORRECT): Maps words to numerical vectors in a way that similar words have similar vectors, capturing semantic relationships. Databricks MLlib supports integration with pre-trained models like Word2Vec or GloVe for this purpose.
Using Word Embeddings in Databricks MLlib:
- Load Text Data: Import your text data into a DataFrame.
- Preprocessing: Tokenize and clean the text (e.g., remove stop words).
- Load Pre-trained Word Embeddings: Utilize models from sources like Gensim or spaCy.
- Convert Words to Vectors: Apply the model to transform words into numerical vectors.
- Machine Learning Model: Use these vectors as features in your ML model.
This approach effectively converts textual data into numerical features, enhancing your machine learning models' performance.