Tokenization Demystified: How LLMs Process Text and Why It Matters

A token is the smallest unit of text an LLM processes. It can be a single character, a sub-word, an entire word, or even punctuation and trailing spaces. For English text, a rough rule of thumb is that 1 token equals about 4 characters, or roughly 0.75 words. A 100-word prompt typically becomes 130–140 tokens. The 4-character rule is an approximation; actual token counts vary by language and formatting. The same sentence can split into more tokens than words: 'Learning AI is fun!' becomes six tokens because suffixes, spaces, and punctuation are often treated as separate tokens.

Sub-word tokenization solves the out-of-vocabulary problem. LLMs break unseen words, typos, or new slang into known sub-word pieces. The word 'ungettable' becomes un + get + table. This keeps the vocabulary manageable: a fixed set of roughly 50,000 to 100,000 sub-word tokens handles billions of phrases, instead of a dictionary of millions of whole words. The dominant techniques are byte pair encoding (BPE) and WordPiece. BPE starts from characters and iteratively merges the most frequent adjacent pairs until it reaches a target vocabulary size; WordPiece selects merges by likelihood. Tokenization is deterministic: the same text and model always produce the same token sequence. Because spaces are often baked into tokens, formatting matters. In code, indentation spaces are processed as distinct token fragments, which can significantly inflate token counts in languages like Python.

Token counts directly affect a developer's bottom line. API providers bill per token for both input and output. Context windows are measured in tokens, not words. When a prompt exceeds the window, depending on the provider, the request fails or the tail of your input is dropped, so your final instructions never reach the model. Latency scales with token count as well. A longer prompt delays the first token, because the model must process all input before generating. For a pipeline with 100,000 requests, an extra 10 tokens per request adds a million tokens to your bill.

Start with the 4-character rule to estimate token counts, then validate with tiktoken on your actual data. OpenAI's tiktoken counts tokens for a specific model before you send a request. Use it in your build step so estimates are exact, not guessed. When writing prompts, minimize filler words and avoid verbose phrasing. Unrecognized words split into multiple billed pieces; shorter, common words usually map to a single token.

Multilingual text needs extra care. Tokenizers trained mostly on English may split non-English words into more tokens per word. Test your prompts in each target language and adjust phrasing if the count climbs. For example, the German word 'Lebensversicherungsgesellschaft' may split into multiple sub-word tokens, whereas a common English word like 'the' is a single token. This difference can be substantial: a compound German word can consume as many tokens as a short English sentence. To see the difference in your own text, run the same prompt in English and in your target language through tiktoken and compare the counts. This gives you a concrete number for your specific use case, rather than relying on a rule of thumb. When you embed code in a prompt, compact formatting is a token-saving decision, not a cosmetic one. In agent loops, each iteration re-sends the same prompt, so a 10-token saving per iteration becomes a 10,000-token saving over 1,000 iterations. Loop engineering treats prompts as part of the system, not one-off inputs.

Log token usage per request in production; it catches truncation before it reaches users.

Code produces the most surprising counts. A count from one tokenizer does not transfer to another. That mismatch explains many 'why is my context full' bugs in production. In retrieval-augmented generation, the retrieved chunks consume tokens, so token-efficient retrieval directly affects how much context fits. For instance, with an 8,000-token context window, a 1,500-token prompt leaves only 6,500 tokens for the response and any retrieved context. The same rule applies when you run models locally: check the tokenizer your local runtime uses before trusting a count from an API tool.