How to Estimate Tokens for RAG (and Why Character Counts Mislead)

Models and pricing count tokens, but chunking libraries usually count characters — and the two don't map cleanly. The rough rule is ~4 characters per token for English prose, but it varies with content, code, and language.

Character counts mislead because a 600-character chunk isn't a fixed number of tokens. To budget context and cost accurately, you need the token count, not the character count.

Why tokens, not characters

Everything that matters downstream is measured in tokens: the context window the model can hold, the embedding cost, the generation cost. But the chunking step usually operates on characters, because that's what's easy to split on. This mismatch is a quiet source of surprises — you set a character size, and the token reality turns out different from what you assumed.

The rough rule, and where it breaks

For English prose, a token is roughly four characters — so ~600 characters is ~150 tokens, give or take. It's a useful rule of thumb, but it breaks down in exactly the cases you care about. Code tokenizes differently from prose. Numbers, punctuation, and rare words split into more tokens. Other languages diverge from the English ratio entirely. The rule is a starting estimate, not a guarantee.

~4 characters per token — until it isn't.

Why the mismatch costs you

• Context budget — if you assume 600 characters is fewer tokens than it is, you can overflow the context window you planned.
• Cost estimates — embedding and generation are priced per token, so a character-based estimate can be off by a wide margin.
• Retrieval tuning — 'top-k = 5 chunks' means very different token loads depending on real chunk token sizes.

Estimating tokens without the exact tokenizer

The precise answer comes from the exact tokenizer your model uses. But for planning, a good subword estimate — one that accounts for word length, punctuation, and numbers rather than just dividing characters by four — tracks real tokenizers closely enough to budget confidently. The point is to get away from raw character counts, which are the least accurate signal.

See tokens per chunk

The RAG Chunk Visualizer estimates tokens for every chunk and the whole document, using a subword heuristic rather than a crude character divide. You can see immediately whether your chunks land near your token target and how the total maps to embedding and generation cost — the character-to-token guesswork removed.