Answer: Decrease temperature to 0.2

## Explanation When an LLM generates overly long and repetitive answers, it's often sampling from too broad a distribution of possible next tokens. The goal is to make the model more deterministic and focused on the most probable tokens. ### Analysis of Options: **A. Increase top-p to 1.0** - This would make the sampling more diverse, not less. Top-p (nucleus sampling) with p=1.0 means sampling from the entire vocabulary, which would increase randomness and potentially make outputs more verbose. **B. Decrease temperature to 0.2** - **CORRECT**. Temperature controls the randomness of predictions: - Lower temperature (closer to 0) makes the model more deterministic and confident in its predictions - The model will sample more from the highest probability tokens - This reduces randomness and makes outputs more focused and concise **C. Increase top-k to 200** - This would increase the number of tokens considered for sampling, making the output more diverse and potentially more verbose. **D. Use temperature = 1.0 and top-p = 0.95** - Temperature=1.0 is the default setting, and top-p=0.95 allows sampling from a broad distribution. This would maintain or increase randomness, not reduce it. ### Why Decreasing Temperature Works: - Temperature scaling modifies the probability distribution before sampling - Lower temperature sharpens the distribution, making high-probability tokens even more likely - This reduces the chance of sampling less likely tokens that might lead to repetition or verbosity - The model becomes more conservative and sticks to the most probable continuations For reducing verbosity and repetition while maintaining coherence, decreasing temperature is the most direct approach.

Author: Ritesh Yadav