CMSManhattan / JiRack-Ultra-Tokenizer-256K

Enjoy — We extend the JiRack Models Ecosystem! 🚀

JiRack Utra Tokenizer 256

JiRack Utra Tokenizer - 347 active language editions of Wikipedia

High-performance production-grade Byte-Level BPE tokenizer developed as part of the JiRack Ternary Models ecosystem.

This is the Ultra version designed for maximum quality, better compression vs 128k Pro version, and precision in complex real-world applications.

• JiRackTernary_1b model https://huggingface.co/kgrabko/JiRackTernary_1b

Open Robot platform

• Tiangong : https://english.www.gov.cn/english.www.gov.cn/news/202411/13/content_WS673406e2c6d0868f4e8ece33.html
• Unitree g1 https://a.co/d/0e4A8YVc
• LimX Oli https://www.limxdynamics.com/en/products/oli?channel=option_google_advertising__c-
• ubtrobot https://www.ubtrobot.com/en/
• x-humanoid https://www.x-humanoid.com/detail/hskw.html

Key Features

• Algorithm : Byte-Level BPE
• Vocabulary Size : 262,144 tokens — excellent balance between precision and efficiency
• Multilingual & Technical Strength : Optimized for English, Russian, code, scientific literature, and technical documentation
• Domain Specialization : Strong performance on programming languages, engineering, robotics, and scientific texts

Special Tokens Support

• Full ChatML dialogue format ( <|im_start|> , <|im_end|> )
• FIM (Fill-in-the-Middle) support for code generation
• Rich set of domain routing tokens ( __CODING__ , __PYTHON__ , __ROBOTICS__ , __SCIENCE__ , etc.)
• Extended robotics and control tokens

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("CMSManhattan/JiRack-Ultra-Tokenizer-256K")

text = "__CODING__ __PYTHON__ Write a merge sort function in Python."

tokens = tokenizer.tokenize(text)
token_ids = tokenizer.encode(text)

print("Tokens:", tokens)
print("Token IDs:", token_ids)

Vocab size: 262144
pad_token_id: 2
eos_token_id: 1

JiRack Pretrain Dataset

https://huggingface.co/datasets/CMSManhattan/JiRack-Pretrain-Dataset

python train_jirack_accelerate.py
Processing jirack_pretrain_chunk_0.pt:  27%|█████████████████████▋                                                  | 268/1000 [22:02:45<60:12:33, 296.11s/it, loss=6.3145, avg_loss=7.0132, ppl=1111.16]
Processing jirack_pretrain_chunk_0.pt:  87%|███████████████████████████████████████████████████████████████████████      | 866/1000 [75:57:47<13:13:29, 355.29s/it, loss=2.8616, avg_loss=5.9877, ppl=398.52]

Benchmark for tokens quality .

(venv_ji) root@jirack2:/mnt/nfs_share/JiRackTokenizerPro_256K# python test_v1.py
=== Text after ChatML Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Write a merge sort function in Python.<|im_end|>


=== Tokens (IDs) ===
[5, 454, 3233, 943, 522, 21130, 109922, 6440, 269, 4, 454, 6, 454, 73, 476, 88, 27953, 522, 60261, 6087, 1810, 576, 6530, 269, 4, 454]

=== Decoding Token by Token ===
5 -> '<|im_start|>system'
454 -> '
'
3233 -> 'You'
943 -> ' are'
522 -> ' a'
21130 -> ' precise'
109922 -> ' router'
6440 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
27953 -> ' Write'
522 -> ' a'
60261 -> ' merge'
6087 -> ' sort'
1810 -> ' function'
576 -> ' in'
6530 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== Текст после ChatML шаблона ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Напиши функцию сортировки слиянием на python.<|im_end|>


=== Токены (ID) ===
[5, 454, 3233, 943, 522, 21130, 109922, 6440, 269, 4, 454, 6, 454, 73, 476, 88, 33218, 51217, 18533, 70799, 61781, 159246, 2230, 7940, 1078, 1135, 1227, 917, 57473, 269, 4, 454]

=== Декодирование по токенам ===
5 -> '<|im_start|>system'
454 -> '
'
3233 -> 'You'
943 -> ' are'
522 -> ' a'
21130 -> ' precise'
109922 -> ' router'
6440 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
33218 -> ' Нап'
51217 -> 'иши'
18533 -> ' функ'
70799 -> 'цию'
61781 -> ' сор'
159246 -> 'тиров'
2230 -> 'ки'
7940 -> ' сл'
1078 -> 'ия'
1135 -> 'ни'
1227 -> 'ем'
917 -> ' на'
57473 -> ' python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== ChatML 模板处理后的文本 ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ 用 Python 写一个归并排序函数。<|im_end|>


=== Token (ID) ===
[5, 454, 3233, 943, 522, 21130, 109922, 6440, 269, 4, 454, 6, 454, 73, 476, 88, 190845, 6530, 476, 24555, 16049, 57962, 14098, 19575, 26588, 88529, 882, 4, 454]

=== 逐个 Token 解码 ===
5 -> '<|im_start|>system'
454 -> '
'
3233 -> 'You'
943 -> ' are'
522 -> ' a'
21130 -> ' precise'
109922 -> ' router'
6440 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
190845 -> ' 用'
6530 -> ' Python'
476 -> ' '
24555 -> '写'
16049 -> '一个'
57962 -> '归'
14098 -> '并'
19575 -> '排'
26588 -> '序'
88529 -> '函数'
882 -> '。'
4 -> '<|im_end|>'
454 -> '
'


=== النص بعد تطبيق قالب ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ اكتب دالة فرز بالدمج (merge sort) بلغة بايثون.<|im_end|>


=== الرموز (IDs) ===
[5, 454, 3233, 943, 522, 21130, 109922, 6440, 269, 4, 454, 6, 454, 73, 476, 88, 35153, 9711, 935, 30958, 199128, 6585, 5673, 1201, 578, 108628, 6087, 264, 6685, 29111, 76156, 2731, 1452, 269, 4, 454]

=== فك الترميز رمزا برمز ===
5 -> '<|im_start|>system'
454 -> '
'
3233 -> 'You'
943 -> ' are'
522 -> ' a'
21130 -> ' precise'
109922 -> ' router'
6440 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
35153 -> ' اك'
9711 -> 'تب'
935 -> ' د'
30958 -> 'الة'
199128 -> ' فرز'
6585 -> ' بال'
5673 -> 'دم'
1201 -> 'ج'
578 -> ' ('
108628 -> 'merge'
6087 -> ' sort'
264 -> ')'
6685 -> ' بل'
29111 -> 'غة'
76156 -> ' باي'
2731 -> 'ث'
1452 -> 'ون'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== Text nach ChatML-Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Schreibe eine Merge-Sort-Funktion in Python.<|im_end|>


=== Token (IDs) ===
[5, 454, 3233, 943, 522, 21130, 109922, 6440, 269, 4, 454, 6, 454, 73, 476, 88, 118471, 18077, 7384, 237233, 268, 124070, 268, 67699, 32813, 576, 6530, 269, 4, 454]

=== Dekodierung Token für Token ===
5 -> '<|im_start|>system'
454 -> '
'
3233 -> 'You'
943 -> ' are'
522 -> ' a'
21130 -> ' precise'
109922 -> ' router'
6440 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
118471 -> ' Schre'
18077 -> 'ibe'
7384 -> ' eine'
237233 -> ' Merge'
268 -> '-'
124070 -> 'Sort'
268 -> '-'
67699 -> 'Fun'
32813 -> 'ktion'
576 -> ' in'
6530 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== Texte après le modèle ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Écris une fonction de tri fusion en Python.<|im_end|>


=== Tokens (IDs) ===
[5, 454, 3233, 943, 522, 21130, 109922, 6440, 269, 4, 454, 6, 454, 73, 476, 88, 4893, 170242, 4842, 56840, 599, 3034, 34974, 714, 6530, 269, 4, 454]

=== Décodage token par token ===
5 -> '<|im_start|>system'
454 -> '
'
3233 -> 'You'
943 -> ' are'
522 -> ' a'
21130 -> ' precise'
109922 -> ' router'
6440 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
4893 -> ' É'
170242 -> 'cris'
4842 -> ' une'
56840 -> ' fonction'
599 -> ' de'
3034 -> ' tri'
34974 -> ' fusion'
714 -> ' en'
6530 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

📧 Contact & Licensing

For joint ventures, hardware integration, or licensing inquiries:

• Email: [email protected]
• Phone: +1 (516) 777-0945
• Location: New York, USA

📧 Copyright 2026 CMS Manhattan . All rights reserved