Tokens Are the Transformer's Real Work Unit
Explain why transformers operate on tokens rather than whole human words.
If you picture a transformer reading words, you start the system one layer too high. The first compression step Before the transformer runs, text is converted into token IDs. That conversion is not a side detail. It defines the sequence length, the vocabulary entries the model can embed, and the units the model will later predict. A token can be a whole word, a word fragment, punctuation, a leading-space pattern, or a technical fragment. Why subwords exist A fixed vocabulary cannot contain every possible name, typo, SKU, and new term. Subword tokenization lets the system represent rare strings by composing…
Sign up free — one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in