A tokenizer is the first thing a language model sees and the last thing anyone optimises. For a model built to serve regulated European industries such as finance, insurance and legal, that is exactly the wrong way round. We built Cadmus, a byte pair encoding tokenizer covering 60 natural languages, and benchmarked it against the tokenizers of the leading open source models. On European languages, it leads every one of them.
Why build our own tokenizer
Three reasons, all of them specific to a European production model rather than a research demo.
European coverage is a first priority, not a side effect. Every official EU language is in the training mix with an explicit byte budget. Most frontier tokenizers are optimised for English and Chinese; their European coverage is whatever falls out of a web crawl. That shows up directly as wasted tokens, and wasted tokens are wasted context, latency and cost on every European request.
Reasoning happens in English, production happens in 27 languages. A modern reasoning model thinks in English chain of thought regardless of the user's language. Cadmus is deliberately built around that asymmetry: English is oversampled for cheap reasoning tokens, while the European languages keep the coverage that customer facing output demands.
Numbers and documents are not an afterthought. Cadmus uses a place value invariant digit scheme, so the ones digit of a number always lands in the same position regardless of length. This is a structural property that matters when a model reads IBANs, ISINs and monetary amounts in free text. The vocabulary also reserves dedicated slots for image, video, audio and OCR markers, so a later multimodal model never needs a disruptive vocabulary extension after the fact.
| Tokenizer | Digit grouping | Tokens | Fixed position |
|---|---|---|---|
| Cadmus (Ablatic) | right to left, in threes | 4 | ✓ |
| DeepSeek V4 | left to right, in threes | 4 | ✗ |
| Kimi K2.6 | left to right, in threes | 4 | ✗ |
| GLM 5.1 | irregular | 7 | ✗ |
| Qwen 3.5 | one per digit | 12 | ✓ |
| Gemma 4 | one per digit | 12 | ✓ |
| MiMo V2.5 Pro | one per digit | 12 | ✓ |
| Mistral Medium 3.5 | one per digit | 12 | ✓ |
European languages: Cadmus vs. the field
The headline metric is bytes per token (BPT): how many bytes of real text each token carries. Higher is better, because more text per token means more context, fewer tokens to generate and lower cost. The table below shows the mean BPT across the 27 European languages on the decontaminated FLORES-200 devtest.
| Tokenizer | Mean BPT | vs. Cadmus |
|---|---|---|
| Cadmus (Ablatic) | 4.512 | · |
| Gemma 4 | 3.634 | +24.2% |
| Mistral Medium 3.5 | 3.612 | +24.9% |
| Qwen 3.5 | 3.574 | +26.2% |
| GLM 5.1 | 3.350 | +34.7% |
| DeepSeek V4 | 3.291 | +37.1% |
| MiMo V2.5 Pro | 3.053 | +47.8% |
| Kimi K2.6 | 3.041 | +48.4% |
“vs. Cadmus” is the additional European text per token Cadmus carries over each comparator. On a per language paired test, Cadmus wins on all 27 European languages against every tokenizer above.
And it still leads on English
A European focus usually costs you English efficiency. It does not here. Cadmus is also the most efficient tokenizer on English in this comparison.
| Tokenizer | English BPT | vs. Cadmus |
|---|---|---|
| Cadmus (Ablatic) | 5.057 | · |
| DeepSeek V4 | 4.919 | +2.8% |
| Kimi K2.6 | 4.902 | +3.2% |
| GLM 5.1 | 4.892 | +3.4% |
| MiMo V2.5 Pro | 4.815 | +5.0% |
| Mistral Medium 3.5 | 4.780 | +5.8% |
| Gemma 4 | 4.730 | +6.9% |
| Qwen 3.5 | 4.661 | +8.5% |
The lead holds on out of distribution text too: on a Wikipedia 2025 set published after every tokenizer's training cutoff, Cadmus keeps the top spot on English and on the European mean.
And across the rest of the world
The pattern holds beyond Europe. Across 33 non-European languages, Cadmus has the highest average bytes per token in the comparison, because it spreads vocabulary evenly across the long tail of mid-resource scripts (African, South East Asian, Caucasus and Cyrillic) that English and Chinese centric tokenizers cover only thinly. The Chinese built tokenizers are strong on Chinese itself, but leave most of this set thinly served.
| Tokenizer | Mean BPT | vs. Cadmus |
|---|---|---|
| Cadmus (Ablatic) | 6.064 | · |
| Gemma 4 | 5.670 | +6.9% |
| Mistral Medium 3.5 | 4.607 | +31.6% |
| Qwen 3.5 | 4.567 | +32.8% |
| DeepSeek V4 | 3.887 | +56.0% |
| Kimi K2.6 | 3.590 | +68.9% |
| MiMo V2.5 Pro | 3.049 | +98.9% |
| GLM 5.1 | 3.005 | +101.8% |
Mean across all 33 non-European languages. Cadmus has the highest mean and the most per-language wins; single-language specialists still lead on the languages they target (for example, the Chinese tokenizers on Chinese).
How it's built
Cadmus is a two phase byte pair encoder. A standard BPE phase learns the bulk of the vocabulary; a second SuperBPE phase adds merges across word boundaries for a further compression gain. A dedicated curriculum reserves merge budget for underrepresented scripts such as Indic, CJK, Arabic and Hebrew, so smaller languages are not crowded out by English and code. English is oversampled 2× over the other 59 languages to match the way modern reasoning models actually reason, and a split minimum length filter keeps both long form prose and short code fragments well tokenised. The tokenizer and its reproduction pipeline are going to be released under the Apache 2.0 licence.
More to come
This is the short version. The full methodology, including the ablation chain, the digit tokenization study, the decontamination analysis and the complete per language results across all 60 languages, is written up in the accompanying paper, which we will publish later this year. More on that then.