Bochkov / bvv241-nemo

bvv241-nemo: SOTA Mistral Nemo Tokenizer with BVV-mapped Frozen Embedding

Tokenizer Description

This tokenizer is based on a hybrid vocabulary:

This tokenizer uses a strictly structured Unicode mapping scheme:

• Plane 0 (0–65535): All single Unicode code points (monograms) are mapped 1:1 to token codes, directly matching standard Unicode BMP.
• Private and unused code ranges (Plane 0 high + supplementary, e.g., 0xE000–0xF8FF and 65536–131071):
  • All multi-character tokens (bigrams, trigrams, SOTA model token strings) are placed exclusively in these ranges.
• This design achieves total, lossless Unicode text coverage, with all multi-symbol tokens isolated above the core Unicode range.
• Tokenizer uses the Mistral-Nemo model's full vocabulary
• Vocabulary size: 131,072 tokens,
• Embedding dimension: 1024.

The associated normalized_embeddings_weights.pt file contains a [vocab_size x embed_dim] matrix of precomputed, L2-normalized, frozen embeddings.
No semantic information is encoded; embeddings remain fixed throughout LM pretraining. No training or adaptation; suitable for plug-and-play use in research on embedding-free semantic emergence and modular LMs.

How to Get Started with the Tokenizer

Use the code below:

from transformers import AutoTokenizer

from huggingface_hub import hf_hub_download

import torch

tokenizer = AutoTokenizer.from_pretrained('Bochkov/bvv241-nemo')

emb_path = hf_hub_download( repo_id="Bochkov/bvv241-nemo", filename="normalized_embeddings_weights.pt" )

embeddings = torch.load(emb_path)