CMSManhattan / JiRack-UltraPro-Tokenizer-512K

Enjoy — We extend the JiRack Models Ecosystem! 🚀

JiRack Utra Pro Tokenizer 512

JiRack Utra Pro Tokenizer - 347 active language editions of Wikipedia

High-performance production-grade Byte-Level BPE tokenizer developed as part of the JiRack Ternary Models ecosystem.

This is the Ultra Pro version designed for maximum quality, advanced compression vs 256k Ultra version for very large models , and precision in complex real-world applications.

• JiRackTernary_1b model https://huggingface.co/kgrabko/JiRackTernary_1b

Open Robot platform

• Tiangong : https://english.www.gov.cn/english.www.gov.cn/news/202411/13/content_WS673406e2c6d0868f4e8ece33.html
• Unitree g1 https://a.co/d/0e4A8YVc
• LimX Oli https://www.limxdynamics.com/en/products/oli?channel=option_google_advertising__c-
• ubtrobot https://www.ubtrobot.com/en/
• x-humanoid https://www.x-humanoid.com/detail/hskw.html

Key Features

• Algorithm : Byte-Level BPE
• Vocabulary Size : 524,288 tokens — excellent balance between precision and efficiency
• Multilingual & Technical Strength : Optimized for English, Russian, code, scientific literature, and technical documentation
• Domain Specialization : Strong performance on programming languages, engineering, robotics, and scientific texts

Special Tokens Support

• Full ChatML dialogue format ( <|im_start|> , <|im_end|> )
• FIM (Fill-in-the-Middle) support for code generation
• Rich set of domain routing tokens ( __CODING__ , __PYTHON__ , __ROBOTICS__ , __SCIENCE__ , etc.)
• Extended robotics and control tokens

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("CMSManhattan/JiRack-UltraPro-Tokenizer-512K")

text = "__CODING__ __PYTHON__ Write a merge sort function in Python."

tokens = tokenizer.tokenize(text)
token_ids = tokenizer.encode(text)

print("Tokens:", tokens)
print("Token IDs:", token_ids)

Vocab size: 524288
pad_token_id: 2
eos_token_id: 1

JiRack Pretrain Dataset

https://huggingface.co/datasets/CMSManhattan/JiRack-Pretrain-Dataset

python train_jirack_accelerate.py
Processing jirack_pretrain_chunk_0.pt:  27%|█████████████████████▋                                                  | 268/1000 [22:02:45<60:12:33, 296.11s/it, loss=6.3145, avg_loss=7.0132, ppl=1111.16]
Processing jirack_pretrain_chunk_0.pt:  87%|███████████████████████████████████████████████████████████████████████      | 866/1000 [75:57:47<13:13:29, 355.29s/it, loss=2.8616, avg_loss=5.9877, ppl=398.52]

Benchmark for tokens quality .

=== Text after ChatML Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Write a merge sort function in Python.<|im_end|>


=== Tokens (IDs) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 31576, 522, 66472, 6176, 2037, 576, 7637, 269, 4, 454]

=== Decoding Token by Token ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
31576 -> ' Write'
522 -> ' a'
66472 -> ' merge'
6176 -> ' sort'
2037 -> ' function'
576 -> ' in'
7637 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== ChatML 模板处理后的文本 ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ 用 Python 写一个归并排序函数。<|im_end|>


=== Token (ID) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 196893, 7637, 476, 24410, 16482, 61950, 14333, 295880, 92252, 870, 4, 454]

=== 逐个 Token 解码 ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
196893 -> ' 用'
7637 -> ' Python'
476 -> ' '
24410 -> '写'
16482 -> '一个'
61950 -> '归'
14333 -> '并'
295880 -> '排序'
92252 -> '函数'
870 -> '。'
4 -> '<|im_end|>'
454 -> '
'


== Texte après le modèle ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Écris une fonction de tri fusion en Python.<|im_end|>


=== Tokens (IDs) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 170797, 3433, 4484, 56203, 595, 3102, 34759, 720, 7637, 269, 4, 454]

=== Décodage token par token ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
170797 -> ' Éc'
3433 -> 'ris'
4484 -> ' une'
56203 -> ' fonction'
595 -> ' de'
3102 -> ' tri'
34759 -> ' fusion'
720 -> ' en'
7637 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== Text nach ChatML-Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Schreibe eine Merge-Sort-Funktion in Python.<|im_end|>


=== Token (IDs) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 115144, 18528, 6806, 256728, 268, 144869, 268, 386798, 592, 576, 7637, 269, 4, 454]

=== Dekodierung Token für Token ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
115144 -> ' Schre'
18528 -> 'ibe'
6806 -> ' eine'
256728 -> ' Merge'
268 -> '-'
144869 -> 'Sort'
268 -> '-'
386798 -> 'Funkt'
592 -> 'ion'
576 -> ' in'
7637 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== النص بعد تطبيق قالب ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ اكتب دالة فرز بالدمج (merge sort) بلغة بايثون.<|im_end|>


=== الرموز (IDs) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 33139, 9120, 360514, 209685, 6513, 5867, 1187, 580, 121802, 6176, 264, 474504, 84721, 2518, 1435, 269, 4, 454]

=== فك الترميز رمزا برمز ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
33139 -> ' اك'
9120 -> 'تب'
360514 -> ' دالة'
209685 -> ' فرز'
6513 -> ' بال'
5867 -> 'دم'
1187 -> 'ج'
580 -> ' ('
121802 -> 'merge'
6176 -> ' sort'
264 -> ')'
474504 -> ' بلغة'
84721 -> ' باي'
2518 -> 'ث'
1435 -> 'ون'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

=== Текст после ChatML шаблона ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Напиши функцию сортировки слиянием на python.<|im_end|>


=== Токены (ID) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 35013, 91654, 492868, 430356, 174599, 302515, 467815, 874, 66017, 269, 4, 454]

=== Декодирование по токенам ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
35013 -> ' Нап'
91654 -> 'иши'
492868 -> ' функцию'
430356 -> ' сорти'
174599 -> 'ровки'
302515 -> ' слия'
467815 -> 'нием'
874 -> ' на'
66017 -> ' python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'

📧 Contact & Licensing

For joint ventures, hardware integration, or licensing inquiries:

• Email: [email protected]
• Phone: +1 (516) 777-0945
• Location: New York, USA

📧 Copyright 2026 CMS Manhattan . All rights reserved