The Hugging Face tokenizer documentation describes the main tokenization approaches used by transformer models.
How does tokenization work?
A tokenizer applies a fixed vocabulary and splitting algorithm to input, then maps each token to a numeric identifier. The model receives token identifiers, not raw words, and converts them into learned representations before processing them through its transformer architecture. Different model families can tokenize the same sentence differently.
Why do token counts matter?
Tokens consume the model’s context window and often determine API cost. Long prompts, code, tables, and multilingual text can use more tokens than a simple word count suggests. Measure with the tokenizer for the model you actually use, especially before setting retrieval chunk sizes or output limits. A token is therefore both a model input unit and a practical capacity constraint, while embeddings turn token sequences into useful numerical representations.