CMSManhattan / JiRack-Router-Tokenizer-65K

Enjoy — We extend the JiRack Models Ecosystem! 🚀

• The Best RAG with Mixture-of-Experts via CMS Manhattan RAG System with JiRack Ternary Experts for CPU Solution.
• JiRackTernary_1b model https://huggingface.co/kgrabko/JiRackTernary_1b

JiRack Router Tokenizer Pro

High-performance custom Byte-Level BPE tokenizer trained on the full Wikipedia dump across multiple languages.

Developed specifically for intelligent routing models and Mixture-of-Experts systems.

Model Details

• Developer : Konstantin Grabko (JiRack)
• License : Apache License 2.0
• Training Data : Full Wikipedia multilingual dump
• Vocabulary Size : 65,536 tokens
• Special Tokens : 128 reserved tokens (including 40+ domain routing tokens)

Key Features

• Correctly placed <|unk|> token at ID 0
• Full native support for ChatML format ( <|im_start|> / <|im_end|> )
• Large set of specialized routing tokens ( __CODING__ , __MATH__ , __PYTHON__ , __SCIENCE__ , etc.)
• Support JiRack Robotics Technology with tags ( <|action_start|> / <|action_end|> )
• Support JiRack vision , images , audio-visual
• Strong multilingual performance

Basic system and dialogue tokens

"<|unk|>", "<|endoftext|>", "<|padding|>", "<|im_start|>", "<|im_end|>",

Core roles

"<|im_start|>system", "<|im_start|>user", "<|im_start|>assistant",

Additional useful tokens

"<|im_start|>tool", "<|im_start|>function",

Reasoning block

"<|im_start|>thought",

Tool calls

"<|tool_call|>", "<|tool_response|>",

Multimodality and audio-visual block

"<|image|>", "<|video|>", "<|sound|>", "<|voice|>", "<|listening|>", "<|vision|>",

Emotional state (Mood)

"<|mood_happy|>", "<|mood_sad|>", "<|mood_angry|>", "<|mood_neutral|>",

--- FIM (Fill-in-the-Middle) tokens from StarCoder ---

"<|fim_prefix|>", "<|fim_middle|>", "<|fim_suffix|>",

Robotics (trajectory/command boundaries)

"<|action_start|>", "<|action_end|>",

Highlights

• Architecture : Byte-Level BPE (Byte-level Byte Pair Encoding) which natively prevents Out-Of-Vocabulary (OOV) tokens.
• Form Factor : Fully wrapped into Hugging Face PreTrainedTokenizerFast with native ByteLevel decoders for clean cyrillic representation.
• Chat Standard : Out-of-the-box support for ChatML formatting ( <|im_start|> / <|im_end|> ).
• Domain Specialization : Pre-baked atomic routing tokens like __CODING__ and __PYTHON__ etc.

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("CMSManhattan/JiRack-Router-Tokenizer-65K")

text = "<|im_start|>user\n__CODING__ __PYTHON__ Write a merge sort function.<|im_end|>"
print(tokenizer.encode(text))