What are AI tokens? The unit language models actually process

What tokenisation actually does

When you send text to a language model, a step called tokenisation splits it into tokens using rules the model was trained with. A token is often a whole short word, but it can also be a common word fragment, a punctuation mark, or a piece of a longer or unusual word split across two or more tokens. As a rough rule of thumb, one token works out to somewhere around three-quarters of a word in typical English text — useful for a ballpark estimate, not a precise conversion, since the exact split depends on the specific words involved and the model’s own tokeniser.

The model never sees your original letters directly; it sees the resulting sequence of token identifiers, reasons over that sequence, and generates its response the same way — one token at a time, each one chosen based on everything that came before it, until the response is complete and reassembled into ordinary text for you to read.

Why the token is the unit that matters

Tokens matter for three practical reasons, all connected. Cost: most providers charge based on the number of tokens processed, covering both what you send in and what the model generates back, so a longer prompt or a longer answer generally costs more, and per-token pricing is a provider-specific detail worth checking in current documentation rather than assuming it stays fixed. Limits: the context window — the total amount of text a model can consider at once, covered in our companion article — is measured in tokens, so a prompt built from tokens rather than words is what actually gets compared against that ceiling. Speed: because a model generates its answer one token at a time, a longer requested answer generally takes correspondingly longer to produce.

Why tokens are not the same as words

It is tempting to treat “tokens” as just a technical name for “words”, but the two diverge in ways that matter in practice:

• Common short words often map to a single token, while longer or rarer words can split across several.
• Punctuation, spaces and formatting characters typically consume tokens of their own, not just the words either side of them.
• A word count and a token count for the same passage are usually close but rarely identical, which is why a token limit is not simply a word limit under another name.

Non-English text and code tokenise differently

Tokenisers are generally trained on a mix of text dominated by English and common programming languages, and that shows up in how efficiently different content tokenises. The same idea written in a language with different script or word structure can take noticeably more tokens to represent than an equivalent English sentence, because the tokeniser has fewer efficient chunks available for it. Source code tokenises differently again — indentation, symbols and repeated syntax patterns behave nothing like ordinary prose. The practical implication is that a token-based limit or cost estimate calculated from English text does not transfer cleanly to a workload that is mostly in another language or mostly code.