Transformer-based LLMs (Large Language Models) generate text using an autoregressive approach where they predict the next token in a sequence based on all previous tokens. The key innovation is the attention mechanism, which allows the model to weigh the importance of different tokens in the input sequence when making predictions. This enables the model to capture long-range dependencies and contextual relationships effectively, rather than simply classifying, copying, or compressing text.