A tokenizer is the first thing a language model sees and the last thing anyone optimises. For a model built to serve regulated European industries such as finance, insurance and legal, that is exactly the wrong way round. We built Cadmus, a byte pair encoding tokenizer covering 60 natural languages, and benchmarked it against the tokenizers of the leading open-source models. On European languages, it leads every one of them.
- +25%
- more European text per token than the closest competitor, Gemma 4
- 27 / 27
- European languages on which Cadmus leads every comparator
- 60
- natural languages in a single vocabulary
Why build our own tokenizer
Three reasons, all of them specific to a European production model rather than a research demo.
European coverage is a first priority, not a side effect. Every official EU language is in the training mix with an explicit byte budget. Most frontier tokenizers are optimised for English and Chinese; their European coverage is whatever falls out of a web crawl. That shows up directly as wasted tokens, and wasted tokens are wasted context, latency and cost on every European request.
Reasoning happens in English, production happens in 27 languages. A modern reasoning model thinks in English chain of thought regardless of the user's language. Cadmus is deliberately built around that asymmetry: English is oversampled for cheap reasoning tokens, while the European languages keep the coverage that customer-facing output demands.
Numbers and documents are not an afterthought. Cadmus groups digits from the right in threes, a rule implemented directly in the pre-tokenizer, so the ones digit of a number always lands in the same position regardless of length. No separator characters are ever inserted into or removed from the text at runtime. This structural property matters when a model reads IBANs, ISINs and monetary amounts in free text. The vocabulary also reserves dedicated slots for image, video, audio and document markers, so a later multimodal model never needs a disruptive vocabulary extension after the fact.
| Tokenizer | Digit grouping | Tokens | Fixed position |
|---|---|---|---|
| Cadmus (Ablatic) | right to left, in threes | 4 | ✓ |
| DeepSeek V4 | left to right, in threes | 4 | ✕ |
| Kimi K2.6 | left to right, in threes | 4 | ✕ |
| GLM 5.1 | irregular | 7 | ✕ |
| Qwen 3.5 | one per digit | 12 | ✓ |
| Gemma 4 | one per digit | 12 | ✓ |
| MiMo V2.5 Pro | one per digit | 12 | ✓ |
| Mistral Medium 3.5 | one per digit | 12 | ✓ |
Tokens needed to encode a 12-digit number, and whether the ones digit keeps a fixed position regardless of the number's length.
European languages: Cadmus vs. the field
The headline metric is bytes per token (BPT): how many bytes of real text each token carries. Higher is better, because more text per token means more context, fewer tokens to generate and lower cost. The table below shows the mean BPT across the 27 European languages on the FLORES-200 devtest.
| Tokenizer | Mean BPT | vs. Cadmus |
|---|---|---|
| Cadmus (Ablatic) | 4.611 | · |
| Gemma 4 | 3.699 | +24.6% |
| Qwen 3.5 | 3.635 | +26.9% |
| Mistral Medium 3.5 | 3.588 | +28.5% |
| GLM 5.1 | 3.330 | +38.5% |
| DeepSeek V4 | 3.270 | +41.0% |
| MiMo V2.5 Pro | 3.034 | +52.0% |
| Kimi K2.6 | 3.022 | +52.6% |
Mean bytes per token across 27 European languages (FLORES-200 devtest; the 24 official EU languages plus Catalan, Icelandic and Norwegian). Higher is better. “vs. Cadmus” is the additional European text per token Cadmus carries over each comparator; the closest is Gemma 4 at +24.6%. On a per-language paired comparison, Cadmus wins on all 27 European languages against every tokenizer above.
And it still leads on English
A European focus usually costs you English efficiency. It does not here. Cadmus is also the most efficient tokenizer on English in this comparison; its closest competitor on English, DeepSeek V4, carries 9.8% less text per token.
| Tokenizer | English BPT | vs. Cadmus |
|---|---|---|
| Cadmus (Ablatic) | 5.365 | · |
| DeepSeek V4 | 4.885 | +9.8% |
| Gemma 4 | 4.872 | +10.1% |
| Kimi K2.6 | 4.868 | +10.2% |
| GLM 5.1 | 4.859 | +10.4% |
| Qwen 3.5 | 4.798 | +11.8% |
| MiMo V2.5 Pro | 4.782 | +12.2% |
| Mistral Medium 3.5 | 4.747 | +13.0% |
English bytes per token (FLORES-200 devtest). Higher is better.
The lead holds on out-of-distribution text too: on a Wikipedia 2025 set published after every tokenizer's training cutoff, Cadmus keeps the top spot on English and on the European mean.
And across the rest of the world
The pattern holds beyond Europe. Across 33 non-European languages, Cadmus has the highest average bytes per token in the comparison, because it spreads vocabulary evenly across the long tail of mid-resource scripts (African, South East Asian, Caucasus and Cyrillic) that English and Chinese centric tokenizers cover only thinly. The Chinese built tokenizers are strong on Chinese itself, but leave most of this set thinly served.
| Tokenizer | Mean BPT | vs. Cadmus |
|---|---|---|
| Cadmus (Ablatic) | 6.182 | · |
| Gemma 4 | 5.793 | +6.7% |
| Qwen 3.5 | 4.642 | +33.2% |
| Mistral Medium 3.5 | 4.596 | +34.5% |
| DeepSeek V4 | 3.879 | +59.4% |
| Kimi K2.6 | 3.584 | +72.5% |
| MiMo V2.5 Pro | 3.040 | +103.3% |
| GLM 5.1 | 2.996 | +106.4% |
Mean bytes per token across 33 non-European languages (FLORES-200 devtest). Higher is better. Cadmus has the highest mean and the most per-language wins, 6.7% ahead of the closest competitor on this set, Gemma 4; single-language specialists still lead on the languages they target, for example the Chinese tokenizers on Chinese.
How it is built
Cadmus is a two-phase byte pair encoder. A standard BPE phase learns the bulk of the vocabulary; a second SuperBPE phase adds merges across word boundaries for a further compression gain. The resulting vocabulary holds 240,339 tokens covering 60 natural languages and 19 programming languages. A boundary-fix pre-tokenizer keeps code and punctuation-dense text from fragmenting, and the right-to-left digit grouping is a rule of the pre-tokenizer itself, so no separator characters ever enter the text. English is deliberately mixed with chain-of-thought data to match the way modern reasoning models actually reason, while the European languages keep the coverage that customer-facing output demands. A split minimum length filter keeps both long-form prose and short code fragments well tokenised.
More to come
This is the short version. The full methodology, including the ablation chain, the digit tokenization study, the decontamination analysis and the complete per-language results across all 60 languages, is written up in the accompanying paper, which we will publish later this year. More on that then.