Persian token tax: what non-English LLM apps cost

Persian needed 1.53× the English tokens on GPT-4o's tokenizer and 3.26× on GPT-4's, across 1,012 parallel sentences. Method, results and what to test.

Persian (Farsi) text costs more tokens than English for the same content. That premium is the Persian token tax, and its size depends more on the tokenizer than on the script. I counted tokens for the same 1,012 sentences in English and Persian from the FLORES-200 benchmark. Persian needed 1.53× the English count on o200k_base, the encoding OpenAI's tiktoken library maps to GPT-4o, and 3.26× on cl100k_base, the one it maps to GPT-4. Those two were the extremes across the four tokenizers used by text-generation models; bge-m3, an embedding model's tokenizer, came in at 1.10×. Claude and Gemini are not in this measurement: Anthropic does not publish Claude's tokenizer, and I did not test the local tokenizer Google's SDK offers for Gemini. Both vendors count tokens for their own models through an API.

These numbers are a dated snapshot, measured on 17 September 2026 with pinned tokenizer versions.

Why Persian (Farsi) uses more tokens: bytes, merges and training data

Language model APIs charge by the token. In 2023 Ahia and colleagues found evidence that "speakers of a large number of the supported languages are overcharged while obtaining poorer results." Petrov and colleagues showed that the same text translated into different languages can differ in tokenization length "up to 15 times in some cases". A 2025 study of more than two hundred languages reports "often 3-5 times higher RTC ratios" for non-Latin and morphologically complex languages, where RTC is the authors' "Relative Tokenization Cost".

That is the token tax: the same meaning, a bigger bill.

Bytes are where every byte-level tokenizer starts

A byte-level BPE (byte-pair encoding) tokenizer, the kind behind GPT-4o, Qwen3 and DeepSeek-V3, starts from the bytes of the text's UTF-8 encoding and merges frequent byte sequences into tokens. An unaccented Latin letter is one byte; Yoruba's tone-marked and underdotted letters take two or more. A Persian letter is two, and the zero-width non-joiner used inside Persian words is three. On FLORES, the Persian file took 1.69× the bytes of the English file in UTF-8, with 0.94× the characters. Persian is not longer in characters. It is heavier to encode.

What happens next depends on the merges a tokenizer learned from its training data. Moving from cl100k_base to o200k_base cut Persian tokens by 53.5% and English tokens by 1.1%. The newer vocabulary changed Persian, not English.

Yoruba, a Latin-script language, pays more

If script decided the tax, Latin-script languages would barely pay it. A workshop paper by Roy, Roy and Patel (accepted at an IJCAI 2026 workshop, on arXiv since August 2026) found that "Yoruba, despite using the Latin script, exhibits the highest GPT-4o tokenization premium" in their audit, at 2.37×. The authors point to diacritics and vocabulary coverage. In that paper Yoruba is an exploratory comparison: its translations were machine-generated and manually inspected, while Bengali and Hindi are the validated cases.

My run points the same way. Yoruba cost more than Persian on four of the five tokenizers, and Arabic cost less than Persian on four of the five. The Arabic exception is bge-m3, an embedding model built on XLM-RoBERTa and its vocabulary. XLM-R was trained on a hundred languages drawn from CC-100: 13,259M tokens of Persian against 2,869M of Arabic, and Yoruba was not among them. That is consistent with training-data share mattering more than script. It does not prove it.

Results: Persian, Arabic and Yoruba against English

Each cell is a language's total tokens divided by the English total on the same tokenizer:

The tokenizer is the biggest variable. Persian's ratio on the worst tokenizer (3.26×) is about three times its ratio on the best (1.10×), and about twice even among the four tokenizers used for generation (3.26× against 1.53×). Qwen3 treats Arabic far better than Persian, 1.63× against 2.58×, so one Arabic-script number does not cover both languages. The cl100k_base result is consistent with earlier published counts: Petrov and colleagues' FLORES-200 lengths, with the dev and devtest splits joined, put Western Persian at 3.28 times English, against my 3.26× on the devtest split alone.

The totals behind those ratios, with the size of each language file before any tokenizer touched it:

Your corpus will give a different number

Roy and colleagues' audit measured o200k_base on 120 Python-debugging items translated from English, averaging per item. It reports Arabic at 1.44× and Yoruba at 2.37×, both from machine translations. On FLORES prose, which people translated, o200k_base gave Arabic 1.38× and Yoruba 2.17×. Same tokenizer, different text and a different translation process, different result. Count your own prompts, retrieved passages and answers before you budget.

How I measured Persian token counts

A genuinely parallel corpus

The NLLB team describes FLORES-200 as "a human-translated benchmark": the same sentences, in the same order, in every language. I used its devtest split, licensed CC BY-SA 4.0, from the official archive with its SHA-256 hash checked: 1,012 sentences from Wikinews, Wikivoyage and Wikibooks, in English, Western Persian, Modern Standard Arabic and Yoruba.

Parallel matters more than size. An adapted translation, such as a bilingual CV, adds, drops and rephrases content, so its ratio measures the editing as much as the tokenizer. FLORES has its own limit: it is news, travel-guide and textbook prose, not chat, code or contracts.

Open tokenizers, pinned revisions, and why Claude and Gemini are absent

Five tokenizers, all downloadable without an account:

• tiktoken 0.14.0: cl100k_base and o200k_base. tiktoken maps GPT-4 and GPT-3.5 Turbo to the first, and GPT-4o, o1, o3, GPT-4.1 and GPT-5 to the second.

• Qwen3 (Qwen/Qwen3-8B) and DeepSeek-V3 (deepseek-ai/DeepSeek-V3), both byte-level BPE, loaded with Hugging Face tokenizers 0.23.2 at pinned Hub revisions.

• bge-m3 (BAAI/bge-m3), an embedding model whose tokenizer is a SentencePiece Unigram model with the XLM-R vocabulary, not byte-level BPE. Unigram is a different algorithm: it picks the likeliest split of the text from a fixed vocabulary. It matters for retrieval, not generation.

Each sentence was encoded on its own, with no special tokens, and the counts were summed. Per-sentence medians tell the same story as the totals. The core fits in a few lines:

# after: tar -xzf flores200_dataset.tar.gz
import tiktoken
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer


def sentences(lang):
    with open(f"flores200_dataset/devtest/{lang}.devtest", encoding="utf-8") as f:
        return f.read().rstrip("\n").split("\n")


english, persian = sentences("eng_Latn"), sentences("pes_Arab")

o200k = tiktoken.get_encoding("o200k_base")
qwen3 = Tokenizer.from_file(
    hf_hub_download(
        "Qwen/Qwen3-8B",
        "tokenizer.json",
        revision="b968826d9c46dd6066d109eabc6255188de91218",
    )
)


def o200k_tokens(texts):
    return sum(len(o200k.encode_ordinary(t)) for t in texts)


def qwen3_tokens(texts):
    return sum(len(qwen3.encode(t, add_special_tokens=False).ids) for t in texts)


print(round(o200k_tokens(persian) / o200k_tokens(english), 2))
print(round(qwen3_tokens(persian) / qwen3_tokens(english), 2))

The measurement record, enough to repeat the run exactly:

corpus: https://dl.fbaipublicfiles.com/nllb/flores200_dataset.tar.gz
sha256: b8b0b76783024b85797e5cc75064eb83fc5288b41e9654dabc7be6ae944011f6
split: devtest
measured: 2026-09-17
libraries:
  tiktoken: 0.14.0
  tokenizers: 0.23.2
  huggingface_hub: 1.31.0
encodings: [cl100k_base, o200k_base]
hub_revisions:
  Qwen/Qwen3-8B: b968826d9c46dd6066d109eabc6255188de91218
  deepseek-ai/DeepSeek-V3: e815299b0bcbac849fa540c768ef21845365c9eb
  BAAI/bge-m3: 5617a9f61b028005a4858fdac845db406aefb181
method: each sentence encoded on its own, no special tokens, counts summed per language

Claude and Gemini are absent on purpose. Anthropic does not publish Claude's tokenizer, so any Claude ratio I gave would be a guess with a decimal point. Google's Gen AI SDK for Python ships an experimental, text-only local tokenizer that maps some Gemini models to open Gemma tokenizers; I did not measure it. Both vendors count tokens for you. Anthropic's count_tokens endpoint is free but rate-limited, and its documentation calls the result "an estimate"; Gemini's count_tokens counts the input. Claude 4.7 and later models use a newer tokenizer, on which the same input text produces "approximately 30 percent more tokens than on earlier models". A token count belongs to a model, not a vendor.

Bounds that hold, and one that does not

For byte-level BPE tokenizers that split text into pre-tokens before merging, as all four here do, two limits frame the count. Tokens cannot outnumber the bytes of the text in UTF-8, as the tokenizer normalises it, because every token covers at least one byte. Nor can they be fewer than the tokenizer's own pre-tokens, the chunks its splitting pattern cuts the text into, because merges never cross a pre-token boundary. I checked both bounds sentence by sentence on the four byte-level BPE tokenizers and all four languages: 16,192 checks, 0 violations.

The tokenizer playground on this site shows both numbers for any text you type, computed in the browser: the cl100k_base pre-token split and the text's bytes in UTF-8. For cl100k_base, the true count lies between them, which is what the caption under each pane says.

The lower bound belongs to one tokenizer. The cl100k_base split is not a floor for o200k_base: on 28 English sentences, o200k_base needed fewer tokens than cl100k_base produces pre-tokens. Neither bound applies to bge-m3's Unigram model.

How the token tax affects price, latency, context and rate limits

Price

API vendors charge for the tokens a model processes and generates. At the same per-token price, the same content in Persian costs about half as much again as English on o200k_base, and more than three times as much on cl100k_base. When both the prompt and the answer are Persian, the extra cost applies to both sides of the bill.

Latency

Output tokens set the pace. OpenAI's latency guide says that "cutting 50% of your output tokens may cut ~50% of your latency". Trimming the prompt matters far less: "cutting 50% of your prompt may only result in a 1–5% latency improvement". A Persian answer that needs more tokens than its English equivalent takes longer to stream. That is vendor guidance applied to the ratios above, not a latency measurement of mine.

Context window

A fixed token window holds less Persian. Dividing by the ratio, the same window holds 31% as much Persian content as English on cl100k_base, 39% on Qwen3, 56% on DeepSeek-V3, 65% on o200k_base and 91% on bge-m3, for this corpus. Roy and colleagues' audit makes the same point for Bengali, whose 1.56× premium on GPT-4o reduces "a nominal 128k-token context window to an effective 82k-token English-equivalent capacity". In a RAG prompt, that means room for fewer retrieved passages.

Rate limits

Rate limits are counted in tokens too. Claude's API limits are "measured in requests per minute (RPM), input tokens per minute (ITPM), and output tokens per minute (OTPM)". Under a tokens-per-minute ceiling, a workload that needs more tokens per request, as Persian did on every tokenizer measured here, hits the limit after fewer requests. For most Claude models, cache reads do not count toward the input limit.

Normalisation that saves tokens, and normalisation that breaks recall

Two features of written Persian look like cheap token savings. Both are traps.

ZWNJ (نیم‌فاصله) is a character, not a space

U+200C ZERO WIDTH NON-JOINER has the Unicode general category Cf, a format control, not whitespace. It stops the letters on either side from joining, so می‌شود reads as one word with a visible break. The FLORES Persian file uses it 1,739 times, in 735 of its 1,012 sentences.

Replacing every ZWNJ with a space changed Persian token counts by -1.59% on cl100k_base, -1.82% on o200k_base, -1.57% on Qwen3, -8.08% on DeepSeek-V3 and +0.00% on bge-m3. bge-m3 did not move because its normaliser already maps ZWNJ to a space: all 735 changed sentences kept identical token ids.

The saving is small on three tokenizers, larger on DeepSeek-V3, and it changes the text: one word becomes two, and what you store, show or quote back is no longer what the writer wrote. Keep ZWNJ in stored text and decide separately what a matcher treats as equal. This site does exactly that: only the folding used inside comparisons treats ZWNJ as a space.

Arabic yeh and kaf versus Persian yeh and keheh

ARABIC LETTER YEH (U+064A) and ARABIC LETTER FARSI YEH (U+06CC) are different characters, as are ARABIC LETTER KAF (U+0643) and ARABIC LETTER KEHEH (U+06A9). At the start or in the middle of a word they look the same; at the end of a word they differ. A string comparison always sees two different characters. Even FLORES mixes them: its Persian file has 11 Arabic yeh and 22 Arabic kaf beside 9,570 Farsi yeh and 2,957 keheh.

Rewriting every Farsi yeh and keheh in the Persian file as Arabic yeh and kaf changed token counts by +2.20% on cl100k_base, +9.49% on o200k_base, -7.48% on Qwen3, +10.39% on DeepSeek-V3 and +11.95% on bge-m3. A rewrite that saves tokens on one tokenizer costs tokens on the other four.

Recall is the real issue. If stored text and queries use different code points, exact and lexical matching fail silently. This site folds Persian text on write: Arabic yeh and kaf become their Persian forms, and ZWNJ stays. It folds the search query the same way. Without that, a query typed with Arabic yeh looks for a character the index does not contain, and returns nothing.

The general rule is narrower: fold Persian text only, because in Arabic, Arabic yeh and kaf are the correct letters. This site's own retriever does not make that distinction yet; it applies the Persian folding to content and questions in every language, Arabic included. The article on hybrid search for Persian RAG explains that gap and covers the rest of the retrieval side.

Persian digits

Writing ASCII digits as Persian digits changed token counts by +0.48% to +1.47%. That is not a reason to change what readers see.

How to reduce Persian token costs: test, not assume

Test each of these on your own traffic before you rely on it.

• Choose the model with your own text. On identical sentences the Persian-to-English ratio ran from 1.10× to 3.26×.

• Count with the provider's endpoint. For Claude and Gemini, the ratios measured here tell you nothing.

• Re-count when the model changes. A newer tokenizer can move every budget you have set.

• Size chunks and context budgets in tokens, per language. A character limit tuned on English holds more tokens in Persian: from about a sixth more on bge-m3 to about three and a half times as many on cl100k_base.

• Cap output length. Output tokens drive latency and cost. Anthropic suggests asking for a number of sentences or paragraphs rather than a word count.

• Cache repeated context, then measure the hit rate. Any saving depends on how often your prefix actually repeats.

• Treat an English system prompt as a hypothesis. It takes fewer tokens, but I have no measurement of what it does to Persian answers.

• Do not strip ZWNJ or switch letterforms to save tokens. The savings are small or inconsistent; the damage to text and matching is not.

How to budget for Persian tokens

A characters-per-token rule is a guess per tokenizer

Persian ran from 1.40 characters per token on cl100k_base to 3.68 on bge-m3, with o200k_base at 3.01. English ran from 4.30 to 4.91. Google's Gemini documentation says "a token is equivalent to about 4 characters" and pairs that with a count of English words: an English-shaped rule, not a statement about Persian.

A reservation of one token per two Persian characters sounds generous. This site used to make it, which is why I tested it on these sentences: it came in below the true count for all 1,012 Persian sentences on cl100k_base and for 976 on Qwen3. It held on o200k_base and bge-m3, and fell short on 6 sentences on DeepSeek-V3. A rule that is safe on one tokenizer can be optimistic on the next, so this site no longer runs one.

A hard ceiling: the byte count

For byte-level BPE, the byte count of the text as the tokenizer normalises it is a hard ceiling, but a loose one. On a scale from 0 at the pre-token count to 1 at the byte count, English sat at 0.015 to 0.017: nearly every English pre-token, roughly a word, is a single token. Persian sat at 0.07 on o200k_base and 0.31 on cl100k_base: split into pieces, but nowhere near into bytes. Reserving bytes therefore over-reserves heavily. That is the right trade for a cap nobody may cross, and the wrong one for a forecast. For Claude and Gemini, count with their endpoints before the call and record the usage they report after it.

Reserve before the call, record after it

The assistant on this site separates those two jobs. Before the language model is called, it adds an estimate for the request (an upper bound on the prompt's tokens, plus the full output cap) to the month's recorded spend and compares the total with a monthly limit. If the spend counter cannot be read, it fails closed. At the limit the model is not called, and the reader gets a notice instead of an error. After the call it records the usage the provider reports. The estimate decides only whether a request may start; the running total comes from the provider's token counts.

Both halves have limits, in opposite directions. The pre-flight estimate counts the prompt as its length in UTF-8 bytes: the ceiling from the section above, not a forecast. Every vocabulary the assistant can be pointed at is a byte-level BPE, so that ceiling holds whatever model it is talking to and whatever language the visitor writes in. For Persian that reserves several times what the prompt will really cost, held for one request and never recorded as spend. It used to count the prompt with the flat two-characters-per-token rule tested above, which came in below the true count on all 1,012 Persian sentences on cl100k_base. A reservation that is too small is a cap wrong in the one direction a cap must never be wrong in, so this measurement is why it changed: the docstring on the estimator cites this same 1,012-sentence run. The monthly total, either way, is built from recorded usage, not from the estimate.

The record has to cope with a silence. OpenAI and Anthropic both report output tokens after the generated text, and OpenAI's streaming reference warns that if a stream is interrupted or cancelled, "you may not receive the final usage chunk". A stream cut off early, by a streaming guardrail or a closed tab, ends before the report arrives — and every character it did produce is on the invoice. So the assistant keeps the text as it streams, and where the provider reported nothing it fills the gap with the same byte count, which over-records rather than under-records. The provider's own figures win wherever it gave any: a turn that completes records exactly what was measured, never an approximation of it. The RAG hallucination guardrails article shows where this check sits among the assistant's other layers.

The same principle holds for a product. At Booq, where I lead AI engineering, each job shows its price, or its starting estimate, before it runs, and a failed job returns its charge automatically, as the Booq case study describes.

FAQ

Does Persian cost more tokens than English on OpenAI models?

Yes. On the same FLORES-200 sentences, Persian needed 1.53× the English count on o200k_base (GPT-4o, GPT-4.1 and GPT-5 in tiktoken's map) and 3.26× on cl100k_base (GPT-4). Your own traffic will give a different ratio, so count it.

How many tokens does Persian use on Claude or Gemini?

This measurement cannot say. Anthropic publishes no tokenizer for Claude, and Google's Gen AI SDK offers an experimental local count for some Gemini models, built on open Gemma tokenizers, which I did not measure. The ratios above do not carry over to either vendor. Use Anthropic's count_tokens endpoint or Gemini's count_tokens on real requests, and count again when you change model.

Is Arabic cheaper than Persian?

On four of the five tokenizers measured here, yes: on o200k_base, Arabic needed 1.38× the English count against Persian's 1.53×. On bge-m3 it was the other way round, 1.17× against 1.10×, consistent with XLM-R's training data holding far more Persian than Arabic.

Should I remove ZWNJ or switch to Arabic letterforms to save tokens?

No. Replacing ZWNJ with a space changed counts by -1.57% to -8.08% on the byte-level tokenizers and changes the words. Arabic letterforms saved tokens only on Qwen3 and cost +2.20% to +11.95% elsewhere. Both break exact matching unless every query changes the same way.

How many characters is a Persian token?

It depends on the tokenizer: from 1.40 on cl100k_base to 3.68 on bge-m3 in this measurement, against 4.30 to 4.91 for English. A flat rule that is safe on one tokenizer can undercount on another. This site retired its own two-characters-per-token reservation after this run and reserves the byte count instead, and that is the general advice: reserve against a ceiling or a provider count, not a rule of thumb.

Sources

• FLORES-200 README (facebookresearch/flores)

• facebookresearch/flores repository: licences and archive notice

• facebook/flores dataset card (Hugging Face)

• FLORES-200 dataset archive

• No Language Left Behind: Scaling Human-Centered Machine Translation (NLLB Team et al., 2022)

• Measuring the Tokenization Premium: A Cost Audit for Underserved Language Communities (Roy, Roy and Patel, 2026, IJCAI 2026 workshop)

• Language Model Tokenizers Introduce Unfairness Between Languages (Petrov et al., NeurIPS 2023)

• Petrov et al., tokenization lengths on FLORES-200 (tokenization_lengths.csv)

• Petrov et al., script that joins the FLORES-200 dev and devtest splits (compute_tokenizations.py)

• Tokenization Disparities as Infrastructure Bias (Teklehaymanot and Nejdl, 2025)

• Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models (Ahia et al., 2023)

• Unsupervised Cross-lingual Representation Learning at Scale (Conneau et al., XLM-R and CC-100)

• M3-Embedding: BGE-M3 (Chen et al.)

• Model card for BAAI/bge-m3

• Model card for Qwen3

• Model card for DeepSeek-V3

• tiktoken model-to-encoding map (openai/tiktoken)

• Claude documentation: Token counting

• Claude documentation: Rate limits

• Claude documentation: Reducing latency

• Claude documentation: Streaming messages

• OpenAI API documentation: Latency optimization

• OpenAI API reference: Chat Completions streaming events

• Gemini API documentation: Understand and count tokens

• Google Gen AI Python SDK: experimental local tokenizer (local_tokenizer.py)

• Google Gen AI Python SDK: Gemini model to Gemma tokenizer map (_local_tokenizer_loader.py)

• Unicode Character Database: UnicodeData.txt

If you are planning an LLM feature for Persian-speaking users, scope your project.