AIGym / tokenizer-65606

Model Card: OED-Biased BPE Tokenizer

This model card describes a custom-built Byte-Pair Encoding (BPE) tokenizer with a vocabulary intentionally biased towards the Oxford English Dictionary (OED).

Model Details

This tokenizer was developed to create a vocabulary that is robust for general text while ensuring the inclusion of a specific, high-value set of words from the OED. It is a subword tokenizer based on the BPE algorithm.

• Model Type: Byte-Pair Encoding (BPE)
• Vocabulary Size (Target): 50,000
• Vocabulary Size (Final): 65,606
• Special Tokens:
  • <|UNK|> (Unknown Token)
  • <|BOS|> (Beginning of Sequence)
  • <|EOS|> (End of Sequence)
  • <|PAD|> (Padding Token)
  • <|MASK|> (Mask Token)
• Normalization: Unicode NFKC
• Pre-tokenization: Regex-based splitting to handle contractions, words, numbers, and whitespace.
  • Regex: (?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+

Training

The tokenizer was trained in a two-stage process designed to bias the vocabulary.

Stage 1: Initial BPE Training

The tokenizer was first trained on a corpus of text from the following Hugging Face datasets:

• motionlabs/fineweb-ultra-mini (10,000 samples)
• suolyer/pile_arxiv (10,000 samples)

The training aimed for a vocabulary size of 50,000 with a minimum token frequency of 2.

Stage 2: Vocabulary Augmentation

After the initial training, the vocabulary was augmented by adding a predefined list of 20,000 words from the Oxford English Dictionary ( 20k.txt ). This step ensures that these specific words are included as complete tokens in the final vocabulary, regardless of their frequency in the training data. This "bias_by_post_add_tokens" method resulted in:

• Initial Vocabulary Size: 50,000
• OED Words Added: 20,000
• Final Vocabulary Size: 65,606

The final vocabulary size exceeds the initial target due to the forced addition of the OED word list.

Intended Use

This tokenizer is intended for natural language processing tasks where a strong representation of the English language, including a rich and potentially less common vocabulary, is important. It is well-suited for models that will process literary, academic, or other forms of text where the OED vocabulary is likely to be present.

Limitations and Bias

The primary bias of this tokenizer is its explicit enrichment with words from the Oxford English Dictionary. While this is an intentional feature, it means the tokenizer may be less efficient for texts that are significantly different in character from the training data and the OED wordlist (e.g., code, specialized technical jargon, or informal social media content). The vocabulary is heavily skewed towards the English language.