alakxender / bert-dhivehi-tokenizer-extended

bert-dhivehi-tokenizer-extended

An extended BERT tokenizer built upon bert-base-multilingual-cased , optimized for Dhivehi (Divehi/Thaana script).

Overview

This tokenizer preserves all English and multilingual coverage of the base model, while adding the top 100,000 high-frequency Dhivehi tokens extracted from a large corpus. It ensures robust tokenization for both English and Dhivehi with no regressions.

How It Was Built

• Base model : bert-base-multilingual-cased
• Corpus : ~16.7M lines of cleaned Dhivehi text (one sentence per line)
• Processing steps :
  1. Count word-level tokens in batches of 100,000 lines
  2. Filter out rare tokens (frequency < 5)
  3. Select top 100,000 tokens by frequency
  4. Extend vocab using tokenizer.add_tokens([...])

Usage Example

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("alakxender/bert-dhivehi-tokenizer-extended")

# Tokenization test
text_dv = "ޖެންޑާގެ ސްޓޭޓް އަޒްރާ"
print(tokenizer.tokenize(text_dv))

• English tokenization unchanged
• Dhivehi text now tokenizes into meaningful word-level tokens with zero [UNK]

Intended Uses

• Downstream NER , QA , classification , or masking tasks in Dhivehi
• Fine-tuning of bert-base-multilingual-cased model with extended vocabulary
• Specially useful for projects in Maldivian/Thaana script

Limitations

• Vocabulary capped at 100,000 new tokens ; may miss very rare words
• Tokenization remains word-level , not subword-based—better performance may require further tokenizer retraining

Files

• vocab.txt , tokenizer_config.json , special_tokens_map.json
  → Standard tokenizer files
• Note : No tokenizer.json ; requires BertTokenizer , not Fast , due to manual extension