openeurollm / tokenizer-256k

OpenEuroLLM Tokenizer (256k)

A 262,144-token SentencePiece BPE tokenizer designed for efficient tokenization across all EU official languages and additional European languages. Trained on 173 GB of curated multilingual text from the OpenEuroLLM data catalogue on LUMI HPC.

Key Results

• Best fertility (tokens per word) on 26 out of 38 European languages , outperforming Llama 3.2, Gemma 3, GPT-OSS, EuroLLM, Qwen 2.5, DeepSeek V3, and Mistral v0.3
• Lowest average fertility overall: 2.12 across 38 languages
• Particularly strong on lower-resource EU languages: Lithuanian (31% fewer tokens than Llama), Hungarian (22%), Icelandic (26%), Maltese (25%)
• English fertility (1.58) within 5% of best (GPT-OSS, 1.51) — no trade-off

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("openeurollm/tokenizer-256k")

text = "Hello world! Bonjour le monde. Hej världen!"
ids = tok(text)["input_ids"]
decoded = tok.decode(ids, skip_special_tokens=True)

print(f"Tokens: {len(ids)}")
print(f"Decoded: {decoded}")

Batch encoding

texts = [
    "The quick brown fox jumps over the lazy dog.",
    "Der schnelle braune Fuchs springt über den faulen Hund.",
    "Le rapide renard brun saute par-dessus le chien paresseux.",
]
batch = tok(texts, padding=True, return_tensors="pt")
print(batch["input_ids"].shape)  # (3, max_len)

Special Tokens

Training Details

Data Sources

The training corpus aggregates cleaned/deduplicated text from: C4, FineWeb-2, Nemotron-CC, MADLAD-400, HPLT, FinePDFs, German-Commons, StarCoder, Proof-Pile-2, Cosmopedia-v2, and FineMath.

Languages (37 + English)

EU Official (23): bg, hr, cs, da, nl, et, fi, fr, de, el, hu, ga, it, lv, lt, mt, pl, pt, ro, sk, sl, es, sv

Additional European (14): sq, eu, bs, ca, gl, is, lb, mk, no, ru, sr, tr, uk, cy

Fertility Evaluation

Average tokens per word across 38 European languages (lower = better), evaluated on 200 Wikipedia articles per language:

See Also

• openeurollm/tokenizer-128k — 128k vocab variant (half the vocabulary, ~9% more tokens per word)