Tokens Explained: Why AI Models Count Pieces of Words

Every AI chat tool and API talks about tokens: limits in tokens, prices per million tokens. A token is simply the unit a language model reads and writes.

What a token is

Models don’t see letters or whole words. Their tokenizer splits text into common chunks. Frequent words are one token (“the”, “Excel”), rarer words are split (“reconciliation” might be 3–4 tokens), and spaces and punctuation are usually attached to the next chunk.

As a rough guide for English, 1 token ≈ ¾ of a word, so 1,000 words ≈ 1,300–1,400 tokens. The exact ratio depends on the model’s tokenizer.

Why it varies

• Other languages: Hindi, Tamil and other scripts often need more tokens per word, because tokenizers saw less of them during training.
• Code and numbers: symbols, indentation and long numbers split into many tokens.
• Formatting: tables and JSON add tokens for every bracket and quote.

Why it matters

• Price: APIs charge per input token (what you send) and per output token (what comes back) — output is usually several times more expensive.
• Context window: the maximum tokens a model can consider at once — your prompt, documents and the conversation so far.
• Speed: output is generated token by token, so long answers take longer.

Saving tokens

• Send only the relevant part of a document (see RAG).
• Ask for the format you need — “a 5-row table” — instead of an essay.
• Remove repeated boilerplate from prompts you send thousands of times.

Want to see a tokenizer from the inside? Build one in lesson 2 of Build Your Own LLM.