Did the new tokenizer fix multilingual AI costs? We measured it.

When OpenAI shipped o200k_base with GPT-4o, one of the stated improvements was better handling of non-English text. That mattered commercially, not just technically: providers bill per token, so a tokenizer that splits your language into more pieces charges you more for saying the same thing.

The claim was easy to make and rarely checked. So we checked it. Five texts in five registers, translated into 25 languages, run through both cl100k_base and o200k_base. Every number below is counted, not estimated.

The short answer: yes, but unevenly

The new tokenizer did close the gap — dramatically for some languages and barely at all for others. Hindi went from 5.35× English to 1.67×, erasing 85% of its penalty. Hungarian moved from 2.40× to 2.02×, erasing 27%.

Why some languages were left behind

The languages that improved most were those paying a byte-level penalty: Hindi, Hebrew and Arabic use scripts that cost multiple UTF-8 bytes per character, and a larger vocabulary gives those scripts far more merged pieces to work with. That is a problem you can fix by spending vocabulary on it.

The languages that improved least were those paying a morphological penalty. Hungarian, Finnish and Turkish are agglutinative: they build long words by stacking suffixes, producing word forms so numerous that no fixed vocabulary can cover them. A bigger vocabulary helps at the margin, but it cannot enumerate a combinatorial space.

Byte problems shrink when you buy more vocabulary. Morphology problems do not.

What this costs in practice

For a product handling 500,000 calls a month at 500 English tokens each, priced at $2.50 per million input tokens, the language penalty alone runs from roughly $1,200 a year for Chinese to nearly $10,000 for Greek. None of that buys a better answer.

There is also a version-lag problem that is easy to miss. If your product still runs on a GPT-4 generation model, you are billed at the old tokenizer's rate regardless of what newer models can do. For Turkish that is 2.17× rather than 1.64×; for Greek, 5.42× rather than 2.32×. Migrating models is a pricing decision, not only a capability one.

The full ranking

Method and its limits

Both tokenizers were run over a parallel corpus of five texts per language, covering business email, support reply, product description, instructions and casual chat. Token counts are exact. The corpus is published in full in the repository.

The honest caveat is translation. These are our translations, and a wordier rendering costs more tokens, so the corpus is auditable by design. We also publish the spread across the five registers rather than a single average — Turkish, for instance, ranges from 1.18× on casual chat to 2.09× on formal writing, and quoting only the flattering end of that would be a choice, not a measurement.