Why does AI cost more in Hindi?
Saying the same thing in Hindi takes 1.67× the tokens it takes in English. Since AI providers bill by the token, that is 67% added to every prompt you send — not for better answers, just for the encoding.
The measurement
7 of 25 by token costThe newer tokenizer closed 85% of the gap. That is real progress, but it is not the same as parity, and older models still bill at the old rate.
What that costs a real product
$5,029per year, wasted on encoding
Based on 500K calls a month at 500 English tokens each, priced at $2.50 per million input tokens. Input side only — output length is a property of the answer, not of your language.
Measured across five registers
One sentence would prove nothing, so the same five texts are measured in every language. The penalty ranges from 1.50× to 1.87× depending on what you are writing.
| Text type | Tokens | vs English |
|---|---|---|
| Business email | 56 | 1.87× |
| Support reply | 56 | 1.51× |
| Product description | 61 | 1.79× |
| Instructions | 59 | 1.74× |
| Casual message | 57 | 1.50× |
Why this happens
Characters outside the Latin range cost several UTF-8 bytes before merging begins, and the tokenizer's learned vocabulary holds comparatively few pieces from this script, so words break apart more than English ones do.
Frequently asked questions
Why does Hindi use more tokens?
Tokenizers learn their vocabulary from training text that is overwhelmingly English, so English word-pieces get their own tokens while other languages' word forms are split into smaller fragments. The effect is mechanical, not a judgement about the language.
Can I do anything about it?
Three things help. Write system prompts and instructions in English while keeping user-facing content in Hindi — the model understands the instruction either way. Prefer newer models, whose tokenizers handle non-English text noticeably better. And trim boilerplate, since the penalty applies to every token you send, including the wasted ones.
Is this measured or estimated?
Measured. The real tokenizers are run over five parallel texts in each language and the counts are exact. The only soft input is the translation itself, which is why the corpus is published in full and the spread across all five texts is shown rather than a single flattering average.
Languages that cost about the same
Method: the real BPE tokenizers — cl100k_base for the GPT-4 generation and o200k_base for GPT-4o and later — are run over a parallel corpus of five texts per language. The token counts are exact, not estimated. The translations are ours and are kept in full in the repository, because a wordier rendering would cost more tokens and you should be able to check ours.