bench-labs / GCTokenizer-v1

GCTokenizer-v1

GCT v1 is a corpus-free multilingual tokenizer built from vocabulary consensus across six open frontier tokenizers: DeepSeek, Kimi, Qwen, GLM, Gemma, and Mistral.

• Reference script: tokenizer implementation & inference python script

It provides four deterministic variants:

• GCT-S — 32,768 tokens
• GCT-M — 65,536 tokens
• GCT-L — 131,072 tokens
• GCT-XL — 262,144 tokens

GCT prioritizes tokens independently selected by multiple existing tokenizers. This produces a strong shared Latin/technical vocabulary and universal UTF-8 byte fallback, while sacrificing some compression efficiency and multilingual balance compared with individually corpus-trained tokenizers.

The tokenizer uses no corpus-trained merges and performs exact lossless round-tripping in its reference implementation.

Multilingual Compression

GCT is designed as a general-purpose tokenizer built from vocabulary consensus across multiple large language-model tokenizers.

To characterize its multilingual behavior, GCT was evaluated on 20 languages using the same measurement procedure for each language. The metric reported is tokens per byte ; lower values indicate more compact tokenization.

Measurements below use GCT-L on Wikipedia plain-text extracts (~30 KB UTF-8 per language), encoded with greedy longest-match segmentation and byte fallback.

These measurements are provided as a characterization of GCT's multilingual tokenization behavior, rather than as a comparison or leaderboard against other tokenizers.

Latin-script languages compress well under the shared consensus vocabulary. Non-Latin scripts fall back more often to UTF-8 byte tokens at the S/M/L tiers; GCT-XL recovers substantially more multi-byte pieces for several scripts (see multilingual_compression.json for full S/M/L/XL tables).

Universal Byte Coverage

GCT includes deterministic byte fallback, allowing arbitrary byte sequences to be represented even when no corresponding vocabulary token exists.

The tokenizer has been tested against random byte sequences, arbitrary UTF-8, invalid UTF-8, control bytes, combining marks, emoji, repeated bytes, and all 256 possible byte values.

Across 40,000 randomized round-trip tests (10,000 per vocabulary tier), every input was reconstructed exactly:

decode(encode(x)) == x

40,000 / 40,000 — 100% exact recovery.