AIGym / AIGT-200k

Model Card: AIGT-200k

Model Details

• Model Type: Byte-Pair Encoding (BPE) Tokenizer
• Vocabulary Size: ~200,000
• Developed by: AIGym Community
• Language(s): Multilingual, with a focus on English, German, Ukrainian, Spanish, Russian, and Code.

Model Description

AIGT-200k is a highly efficient, multilingual Byte-Pair Encoding (BPE) tokenizer with a vocabulary of approximately 200,000. It is designed for modern large language models that require a nuanced understanding of conversational structure, multilingual text, and programming languages.

This tokenizer was developed using a hybrid approach. It starts with a large base of predefined tokens, including the top 100,000 most common words, common symbols, and byte-level fallbacks. This foundation is then expanded through BPE training on a diverse corpus of text and code.

A key feature of this tokenizer is its native support for the OpenAI Harmony chat template . This includes a rich set of special tokens for managing conversational flow, tool usage, and role specification, making it ideal for building advanced, agentic AI systems. It also maintains backward compatibility with the popular ChatML format.

The tokenizer includes dedicated special tokens for multimodal applications and code infilling tasks, reserving tokens for future capabilities like image, video, and audio processing.

Motivation and Philosophy

The primary motivation behind AIGT-200k is to move towards a universal, multi-capable tokenizer that can serve as a standard for a wide range of AI applications.

Instead of creating a tokenizer for a specific dataset or a single task, our goal is to establish a common foundation that supports:

• Multilingualism: Seamlessly handling multiple human languages.
• Multimodality: Providing the necessary tokens to integrate text with other data types like images and audio.
• Multi-Chat Templates: Natively supporting both modern (OpenAI Harmony) and established (ChatML) conversational formats.

By creating a versatile and comprehensive tokenizer, we hope to encourage the community to conform towards a shared standard. This will make models more interoperable, simplify fine-tuning, and allow collective efforts to focus on improving a single, powerful tokenization scheme.

Crucially, this tokenizer was designed specifically to tokenize questions, instructions, and agentic interactions , not just to process large, static datasets. Its vocabulary and special tokens are optimized for the dynamic, turn-by-turn nature of conversational AI.

Intended Uses & Limitations

Intended Uses

This tokenizer is optimized for the following applications:

• Advanced Chatbots and Conversational AI: Native support for the Harmony and ChatML formats allows for the development of sophisticated, multi-turn dialogue systems.
• Multilingual Models: Trained on a diverse set of languages to ensure broad linguistic competence.
• Code Generation and Understanding: A significant portion of the training data consists of code, and it includes special tokens for infilling tasks.
• Tool-Using Agents: The Harmony special tokens provide a robust framework for models that interact with external tools and APIs.
• Foundation Models: Serves as a strong vocabulary base for training new large language models from scratch.

Limitations and Bias

• While trained on a multilingual dataset, the tokenizer's performance will be strongest on the languages most represented in the training data (English, German, Ukrainian, Spanish, Russian, and code).
• The training data is sourced from large, unfiltered web crawls (like C4) and code repositories. As such, the tokenizer may have learned to represent societal biases, stereotypes, and potentially offensive language present in that data. Models trained using this tokenizer may generate content reflecting these biases.

How to Use

The tokenizer can be loaded and used directly from the Hugging Face Hub with the AutoTokenizer or PreTrainedTokenizerFast classes.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("AIGym/AIGT-200k")

# Example of encoding an agentic prompt
prompt = "<|start|><|message|><|user|>\nWhat is the weather in London?<|end|>"
encoded = tokenizer(prompt)
print(encoded["input_ids"])

# Example of decoding
decoded = tokenizer.decode(encoded["input_ids"])
print(decoded)

Training Data

The tokenizer was trained on a composite dataset designed for broad coverage. The training process involved two stages:

1. Predefined Token Collection:

   • AIGym/top-100K-words : A dataset of the 100,000 most frequent words.
   • AIGym/common-for-tokens : A collection of common numbers, symbols, emojis, and control characters.
   • A full set of 256 byte-level tokens to handle any unknown characters.

2. BPE Training Corpus: The BPE vocabulary was trained on a large, sampled dataset from the following sources:

   • Code: macrocosm-os/code-parrot-github-code (50,000 samples)
   • Web Text (English): AIGym/motionlabs-fineweb-ultra-mini (10,000 samples)
   • Mathematical Text: AIGym/semran1-finemath_2048 (10,000 samples)
   • Multilingual Web Text (C4):
     • German ( de ): 2,500 samples
     • Ukrainian ( uk ): 5,000 samples
     • Spanish ( es ): 5,000 samples
     • Russian ( ru ): 5,000 samples
     • Multilingual Mix: 5,000 samples
     • English Multilingual Mix: 2,500 samples

Training Procedure

1. Normalization: Text was normalized using NFKC and control characters were removed.
2. Pre-tokenization: A regex-based pre-tokenizer was used to split text based on linguistic patterns, numbers, and symbols.
3. Model: A Byte-Pair Encoding (BPE) model was used.
4. Training: The BpeTrainer from the tokenizers library was used to train the vocabulary on the corpus, building upon the predefined tokens to reach a final size of approximately 200,000.
5. Special Tokens: A comprehensive set of special tokens for control, multimodality, chat templates, and code infilling was added to the final vocabulary. The post-processor is configured to add <|bos|> and <|eos|> tokens to sequences.

Special Token Categories

• Standard: <|unk|> , <|bos|> , <|eos|> , <|pad|>
• ChatML & Harmony Roles: <|system|> , <|user|> , <|assistant|> , <|developer|> , <|tool|>
• OpenAI Harmony Format: <|start|> , <|end|> , <|message|> , <|channel|> , <|call|> , <|return|> , <|constrain|>
• Code Infilling: <|code_prefix|> , <|code_middle|> , <|code_suffix|>
• Multimodality: <|img_start|> , <|img_end|> , <|IMAGE_SOFT_TOKEN|> (and equivalents for audio/video)
• Reserved: 105 reserved tokens for future expansion ( <|category_reserved_i|> )