AWS Certified Cloud Practitioner

Explanation

Correct Answer: A - Tokens provide language-independent representation

Why tokens are used instead of characters:

1. Language Independence: Tokens represent meaningful units of text rather than individual characters. For example, in English, a token might be a word or subword, while in languages like Chinese, tokens represent characters or character combinations that carry meaning.

2. Efficiency: Tokenization allows models to process text more efficiently by breaking it into meaningful units rather than individual characters, which reduces the computational complexity.

3. Semantic Representation: Tokens capture semantic meaning better than individual characters. For instance, the word "unbelievable" as a single token carries more meaning than processing each character separately.

4. Vocabulary Management: Tokenization helps manage vocabulary size by using subword tokenization techniques (like Byte Pair Encoding) that can handle rare words and out-of-vocabulary terms.

Why the other options are incorrect:

• B: Tokens are NOT always shorter than characters - some tokens can be longer than individual characters, especially in languages where single characters represent entire words.

• C: Characters CAN be encoded numerically (using character encoding like UTF-8), so this statement is false.

• D: Tokens do NOT reduce context-window cost to zero - they still consume computational resources, though they are more efficient than character-level processing.

Key Takeaway: Tokenization is fundamental to how modern language models process text, allowing them to work efficiently across different languages while capturing meaningful semantic units.