AxisQuant / IndicPrayog-tokenizer-64k

IndicPrayog-tokenizer-64k

A 64,000-vocabulary Unigram (SentencePiece-style) tokenizer for Hindi, Marathi, and English , selected out of a 6-way ablation and validated with a proxy-LM training run (bits-per-byte on held-out text) — not just intrinsic fertility metrics.

• Algorithm: Unigram (SentencePiece), Metaspace pre-tokenization
• Vocab size: 64,000
• Normalization: NFC (canonical Devanagari matra composition)
• Special tokens: <pad> (0), <eos> (1), <bos> (2), <unk> (3)
• Training mix: 40% Hindi / 35% English / 25% Marathi ( mix_A )
• Selected by: proxy-LM bits-per-byte (BPB) on held-out hi/mr/en text — the winner disagreement rule between intrinsic ranking and BPB never fired (same variant won both).

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("prashantcp8/IndicPrayog-tokenizer-64k")
tok.encode("यह एक परीक्षण वाक्य है।")
# [84, 39, 6019, 7010, 26]
# -> ['▁यह', '▁एक', '▁परीक्षण', '▁वाक्य', '▁है।']  (5 tokens for a 5-word sentence, fertility 1.0)

vs. Sarvam-1 (the Indic gold standard we benchmark against, 68,096 vocab — 4K larger than ours) on the exact same sentence:

from transformers import AutoTokenizer
sarvam_tok = AutoTokenizer.from_pretrained("sarvamai/sarvam-1")
sarvam_tok.encode("यह एक परीक्षण वाक्य है।", add_special_tokens=False)
# [4999, 4551, 19566, 64104, 4432, 67546]
# -> ['▁यह', '▁एक', '▁परीक्षण', '▁वाक्य', '▁है', '।']  (6 tokens — splits "है।" into 2 pieces)

Same sentence: 5 tokens (ours) vs 6 tokens (Sarvam-1) — ours keeps the sentence-final है। (word + punctuation) as one piece, Sarvam-1 splits it. This is a single illustrative example, not the benchmark; see the full FLORES-200 comparison table below.

Note: intended final home is AxisQuant/IndicPrayog-tokenizer-64k , but the cached HF token used for this session is scoped to the gemma-challenge org and the personal prashantcp8 namespace only — it lacks repo.write on AxisQuant . Published here (private) for now; transfer to AxisQuant once org write access is granted.

Methodology

We trained 6 tokenizer variants (A–F) on an identical 1.5 GB byte-budgeted corpus (hi 600 MB / en 525 MB / mr 375 MB, streamed from Sangraha-verified / FineWeb-Edu), varying algorithm, language mix, and pre-tokenization — to isolate which design choices actually help, rather than shipping a single default run.

Evaluation

Primary: FLORES-200 devtest (1,012 parallel sentences, hin_Deva / mar_Deva / eng_Latn ) — parallel text makes cross-language comparison apples-to-apples (Petrov et al. 2023 methodology). Secondary: 2,000 held-out docs/language from Sangraha-verified / FineWeb-Edu.

Six metrics: fertility, compression (bytes/token), parity (Petrov-style fairness score), round-trip fidelity, Devanagari integrity (% of aksharas/conjuncts kept whole), vocab utilization.

Intrinsic results (6 ours + 8 public baselines)

We beat every non-Indic baseline (Gemma-2, Qwen2.5, Mistral, GPT-2) on Hindi/Marathi fertility and parity by a wide margin, and land within ~10% of Sarvam-1 (the Indic gold standard near our vocab size) while using a 4K-smaller vocabulary. Llama-3.1 was dropped (gated repo, license not token-bypassable). IndicBERTv2 's 0.1% round-trip is expected BERT-tokenizer (WordPiece, lossy by design) behavior, not a defect.

Shortlist (top-3 by mean FLORES fertility, gated on 100% round-trip AND Devanagari integrity ≥ variant A): A, C, D. (B and F were rejected on the integrity gate.)

Proxy-LM validation (the real test)

Intrinsic metrics are proxies. To answer "which tokenizer makes a model learn best," each shortlisted candidate trained an identical ~28M-param Llama-style dense LM from scratch (hidden=320, 6 layers, 5 heads, RoPE, SwiGLU, tied embeddings) on an identical 300M-token slice (same source documents, tokenized per-candidate), 3000 steps, same seed, then measured bits-per-byte (BPB) on held-out hi/en/mr text — BPB normalizes by raw bytes, not token count, making it the fair cross-tokenizer comparison metric.

D wins on every language individually, not just on average — confirming the Unigram algorithm's advantage on morphologically rich Devanagari script carries through to actual downstream LM quality, matching the intrinsic-metric ranking (no override needed).

Bugs found and fixed during this ablation

Six real bugs were caught and fixed via direct verification (not just accepting first-pass numbers) before this tokenizer was selected:

1. add_prefix_space=True spurious "Ġ" markers. ByteLevel injected a leading-space marker at every upstream pretokenizer split boundary, not just at string start — corrupting round-trip fidelity around digits/script transitions. Fixed with add_prefix_space=False .
2. UnicodeScripts() silently dropping whitespace-only pretokens. When a lone space pretoken (left behind after Digits() split off a digit) reached UnicodeScripts() in isolation, it was dropped entirely, silently deleting a character. Fixed by running UnicodeScripts() before Digits() in the pretokenizer Sequence .
3. ByteLevel's word-boundary regex splits Devanagari at every matra (the big one). ByteLevel's internal regex ( use_regex=True , default) groups only Unicode category Letter into a "word" — Devanagari matras are category Mark, not Letter — so every Devanagari syllable was pre-split at matra boundaries (भ|ा|रत instead of भारत), making it structurally impossible for BPE to ever merge across a matra. This alone caused ~3.3–5 tokens/word Hindi/Marathi fertility and 40.6% Devanagari integrity. Fixed with a custom Split regex that groups Letter+Mark runs, combined with ByteLevel(use_regex=False) .
4. Space-isolation regression (introduced by the fix above). The first fix regex isolated every space as its own pretoken (unlike GPT-2's convention of attaching a leading space to the following word), so BPE could never merge a space into a word — nearly doubling English fertility (1.26→2.14). Fixed by attaching a leading space to the following word/digit run: " ?[^\s\d]+| ?\d+|\s+" .
5. Eval script's non-NFC-aware round-trip check. All "ours_*" variants showed 96.9% round-trip despite direct spot-checks showing 100%. Root cause: our tokenizer applies NFC normalization by design, so decode(encode(s)) legitimately returns NFC(s) , which can differ as a raw string from non-canonically-composed FLORES source text while being visually/semantically identical. Fixed the eval script to NFC-normalize both sides before comparing — the correct definition of "lossless" for an NFC-normalizing tokenizer.
6. Proxy-LM tied-embedding weight-init bug. nn.Embedding 's PyTorch default init is N(0, 1) ; since the LM head is tied to the embedding, that same std≈1.0 matrix produced logits directly, giving step-0 loss ~307–309 nats instead of the correct ~ln(64000)≈11.07 for a freshly initialized 64K-vocab LM — wasting most of training recovering from a broken init rather than learning real signal, and producing an unreliable first BPB comparison. Fixed with GPT-2/Llama-style N(0, 0.02) init applied to all Linear / Embedding weights. Re-running with the fix confirmed step-0 loss ≈11.1 for all three candidates and changed nothing about the final ranking (D still wins), but the corrected numbers above are the ones that should be trusted.

Known limitation

tokenizers==0.22.2 's UnigramTrainer(byte_fallback=True) silently ignores the byte_fallback kwarg ( Ignored unknown kwargs option byte_fallback ) — so <unk> is not structurally impossible for this Unigram tokenizer as it would be for the BPE variants (A/B/C/E/F use ByteLevel byte-fallback, which IS structurally lossless). In practice, round-trip fidelity measured 100.0% on the full FLORES-200 eval set, so OOV characters were not encountered in testing.

Reproducibility

Full corpus-sampling, training, eval, shortlist, and proxy-LM scripts are included in AxisQuant/IndicPrayog-tokenizer-ablations (all 6 variants + full results), together with the original experiment plan. Fixed seed (1234) throughout; corpus streamed from ai4bharat/sangraha and HuggingFaceFW/fineweb-edu ; eval on haoranxu/FLORES-200 .

The proxy-LM checkpoint that produced the winning BPB result above is published (private) at prashantcp8/IndicPrayog-proxy-lm-D .