Tokens explained: why 1,000 words ≠ 1,000 tokens

By LW Forge — maintainer of LLM Scout · Updated July 6, 2026

Every LLM provider — OpenAI, Anthropic, Google, DeepSeek — bills you per token, yet almost everyone plans their budget in words or pages. That mismatch is the single most common source of wrong cost estimates we see. A 1,000-word document is not 1,000 tokens; in English it's usually around 1,300-1,400 tokens, and in Portuguese or code it can be considerably more. If your estimate assumes one token per word, your invoice will come in 30-80% higher than planned.

What a token actually is

A token is the unit a language model reads and writes. Before your text reaches the model, a component called the tokenizer splits it into chunks drawn from a fixed vocabulary — typically between 50,000 and 200,000 entries, depending on the model family. Frequent words like "the" or "and" get a token of their own. Less frequent words are split into pieces: "tokenization" might become "token" + "ization", and a rare technical term or a typo can shatter into four or five fragments.

The practical consequence: token count depends on what you write, not just how much. Three things reliably inflate it:

• Language. Tokenizer vocabularies are trained mostly on English, so English text compresses best — roughly 0.75 words per token. Portuguese, Spanish and German typically cost 10-25% more tokens for the same content, because accented forms and longer inflections get split more often.
• Formatting and code. JSON, Markdown tables, XML tags and source code are token-expensive: every brace, quote and indent is real billable content. A compact JSON payload can easily use twice the tokens of the equivalent prose.
• Numbers and unusual strings. IDs, hashes, URLs and long numbers don't exist in the vocabulary, so they get chopped into many small pieces.

The rules of thumb that work

For planning purposes, these conversions hold up well for English prose:

For Portuguese, multiply the result by roughly 1.1-1.25. For code or JSON-heavy prompts, it's safer to assume 1.5-2 tokens per "word" and then verify with a real count — paste a representative sample into our token calculator to see the actual number across models before you commit to a budget.

These are planning figures, not billing figures. The only exact count is the one the provider's own tokenizer produces, and each provider's differs slightly: the same paragraph can be a few percent shorter on one model than another. That variance rarely changes a budget decision, but it's one more reason to build estimates with margins instead of exact numbers.

Why providers bill in tokens at all

It can feel like an obscure unit chosen to confuse buyers, but tokens map directly to compute. A model processes text one token at a time: every input token must be read into the model's attention machinery, and every output token requires a full forward pass to generate. Words would be a terrible billing unit — "internationalization" costs the model far more work than "cat", and languages without spaces (Chinese, Japanese) don't even have clean word boundaries. Tokens are the honest measure of the work done.

That's also why output tokens cost 2-5x more than input tokens on most price lists: generating text is sequential and compute-intensive, while reading input can be parallelized. For budgeting, this means the shape of your workload matters as much as its size — a summarization task (long input, short output) and a content-generation task (short input, long output) with identical total tokens can produce very different bills.

Where word-token confusion actually costs money

Three real scenarios where the 1:1 assumption breaks budgets:

1. Document pipelines. "We process 200-page reports" sounds like a words problem, but a 200-page report at ~500 words per page is 100,000 words — roughly 130,000-140,000 tokens, which doesn't even fit in a 128k context window. Check what really fits with the context window calculator and our guide on how many pages fit in a prompt.
2. Chat history. Every turn of a conversation is re-sent as input on the next turn. A chat that "only" exchanges 300 words per turn accumulates thousands of tokens of history that you pay for again on every single request.
3. System prompts. A 2,000-word system prompt is ~2,600 tokens charged on every call. At high volume, trimming it by half can save more money than switching to a cheaper model.

How to get real numbers for your case

Rules of thumb get you a first draft of a budget; real token counts get you one you can defend. Paste representative samples of your actual prompts — not lorem ipsum — into the token calculator, including formatting, then run the resulting token profile through the cost calculator for your provider. Re-check whenever you change prompt structure, output format or language mix: those three quietly move token counts far more than model version upgrades do.