Answer: Bilingual Evaluation Understudy (BLEU) score

For evaluating foundation models (FMs) on text summarization tasks, the **Bilingual Evaluation Understudy (BLEU) score** is the appropriate metric. BLEU is specifically designed for natural language generation (NLG) tasks, including text summarization and machine translation. It works by comparing n-gram overlaps between the model-generated summary and one or more human-written reference summaries, providing a quantitative measure of how closely the generated text matches the reference in terms of content and phrasing. **Why BLEU is optimal:** - **Purpose-built for NLG:** BLEU was developed to evaluate the quality of machine-generated text against human references, making it directly applicable to summarization. - **Widely adopted:** It is a standard metric in research and industry for text generation tasks, ensuring comparability across models and studies. - **Handles multiple references:** It can accommodate multiple reference summaries, which is common in summarization evaluations to account for variability in human summaries. **Why other options are less suitable:** - **A. F1 score:** This is primarily used for classification tasks, especially with imbalanced datasets. It measures precision and recall for categorical predictions, not the quality of generated text sequences. - **C. Accuracy:** This metric is for classification tasks where predictions are either correct or incorrect. It does not capture the nuanced quality of generated summaries, such as fluency or relevance. - **D. Mean squared error (MSE):** MSE is a regression metric that calculates the average squared difference between predicted and actual numerical values. It is irrelevant for text-based tasks like summarization, which involve discrete language outputs. In summary, BLEU is the established metric for text summarization evaluation due to its alignment with NLG objectives, while the other metrics are designed for different problem types (classification or regression).

Author: LeetQuiz Editorial Team