Tokenizer playground
Type anything. The split, the count and the price update as you go.
See how Persian and English tokenise, and what that costs.
What is real here
No BPE vocabulary is downloaded here. A real merge table is several megabytes, and the readers of this site are on networks where that is a stalled page rather than a demo — the API refuses the same download on its own request path, for the same reason. What is shown instead is exact rather than approximate: the published GPT-4 pre-token split, real UTF-8 bytes, and the reservation the API makes before it calls a model. That reservation is the byte count itself, which is why the two numbers match — an upper bound holds in every language, and a rule of thumb does not. Under GPT-4's vocabulary a true token count always sits between the pre-token count and the byte count. How close it sits to the top of that range is what Persian pays for, and what it reserves against.