alakxender / bert-fast-dhivehi-tokenizer-extended

huggingface.co
Total runs: 0
24-hour runs: 0
7-day runs: 0
30-day runs: 0
Model's Last Updated: June 16 2025
token-classification

Introduction of bert-fast-dhivehi-tokenizer-extended

Model Details of bert-fast-dhivehi-tokenizer-extended

BERT Tokenizer Extended for Dhivehi

This is a BertTokenizerFast built by extending the original bert-base-multilingual-cased with a large Dhivehi corpus. It retains full compatibility with English and other languages while adding wordpiece-level support for Dhivehi script.

How to Use
from transformers import BertTokenizerFast

tokenizer = BertTokenizerFast.from_pretrained("alakxender/bert-dhivehi-tokenizer-extended")

text = "ދިވެހި މަޅި ރޯކުރަނީ The quick brown fox"
tokens = tokenizer.tokenize(text)
print(tokens)
Output:
['ދިވެހި', 'މަޅި', 'ރޯ', '##ކުރަނީ', 'The', 'quick', 'brown', 'f', '##ox']
Tokenizer Details
  • Base model: bert-base-multilingual-cased
  • Type: BertTokenizerFast
  • Vocab size: 150,000
  • Trained on: Cleaned Dhivehi monolingual corpus
  • Special tokens: [PAD] , [UNK] , [CLS] , [SEP] , [MASK]
Tokenization Comparison
Language Stock BERT Extended Tokenizer
English Perfect Perfect
Dhivehi UNKs Full Coverage
Clean Vocabulary

All tokens added are frequent (min freq ≥ 5), unused English tokens are preserved to avoid collisions.

Runs of alakxender bert-fast-dhivehi-tokenizer-extended on huggingface.co

0
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs

More Information About bert-fast-dhivehi-tokenizer-extended huggingface.co Model

More bert-fast-dhivehi-tokenizer-extended license Visit here:

https://choosealicense.com/licenses/mit

bert-fast-dhivehi-tokenizer-extended huggingface.co

bert-fast-dhivehi-tokenizer-extended huggingface.co is an AI model on huggingface.co that provides bert-fast-dhivehi-tokenizer-extended's model effect (), which can be used instantly with this alakxender bert-fast-dhivehi-tokenizer-extended model. huggingface.co supports a free trial of the bert-fast-dhivehi-tokenizer-extended model, and also provides paid use of the bert-fast-dhivehi-tokenizer-extended. Support call bert-fast-dhivehi-tokenizer-extended model through api, including Node.js, Python, http.

bert-fast-dhivehi-tokenizer-extended huggingface.co Url

https://huggingface.co/alakxender/bert-fast-dhivehi-tokenizer-extended

alakxender bert-fast-dhivehi-tokenizer-extended online free

bert-fast-dhivehi-tokenizer-extended huggingface.co is an online trial and call api platform, which integrates bert-fast-dhivehi-tokenizer-extended's modeling effects, including api services, and provides a free online trial of bert-fast-dhivehi-tokenizer-extended, you can try bert-fast-dhivehi-tokenizer-extended online for free by clicking the link below.

alakxender bert-fast-dhivehi-tokenizer-extended online free url in huggingface.co:

https://huggingface.co/alakxender/bert-fast-dhivehi-tokenizer-extended

bert-fast-dhivehi-tokenizer-extended install

bert-fast-dhivehi-tokenizer-extended is an open source model from GitHub that offers a free installation service, and any user can find bert-fast-dhivehi-tokenizer-extended on GitHub to install. At the same time, huggingface.co provides the effect of bert-fast-dhivehi-tokenizer-extended install, users can directly use bert-fast-dhivehi-tokenizer-extended installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

bert-fast-dhivehi-tokenizer-extended install url in huggingface.co:

https://huggingface.co/alakxender/bert-fast-dhivehi-tokenizer-extended

Url of bert-fast-dhivehi-tokenizer-extended

bert-fast-dhivehi-tokenizer-extended huggingface.co Url

Provider of bert-fast-dhivehi-tokenizer-extended huggingface.co

alakxender
ORGANIZATIONS

Other API from alakxender

huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated:August 16 2024