Sponsored by PoYo.AI
Toolify Toolify logo
AI Models
emese-tokenizer-32k

emese-tech / emese-tokenizer-32k

Run on huggingface.co
huggingface.co
Total runs: 0
24-hour runs: 0
7-day runs: 0
30-day runs: 0
Model's Last Updated: May 14 2026

Introduction of emese-tokenizer-32k

Model Details of emese-tokenizer-32k

Emese Tokenizer

32,000-vocab SentencePiece Unigram tokenizer optimized for Hungarian language.

Technical Specs
Property Value
Vocab size 32,000
Training corpus Hungarian Wikipedia (1.16M paragraphs, ~252M chars)
Algorithm SentencePiece Unigram (pure EM)
Avg. chars/token 4.90
Avg. tokens/word 1.54
Max piece length 24 chars
Training sample 500,000 paragraphs
Coverage 0.9999
Normalization NFKC
Shrinking factor 0.85
EM sub-iterations 4
Seed sentencepieces 2,000,000
Usage 
import sentencepiece as spm

sp = spm.SentencePieceProcessor(model_file="emese-tokenizer/emese-tokenizer.model")
tokens = sp.encode("Budapest Magyarország fővárosa.", out_type=int)
text = sp.decode(tokens)
Special Tokens
ID Token Purpose
0 <unk> Unknown token
1 <s> Beginning of sequence
2 </s> End of sequence
3 <eos> End of document (used as separator in training data)
Comparison with Other Hungarian Tokenizers
Model Vocab c/tok tok/word Algorithm
Emese 32K 4.90 1.54 Unigram (Pure EM)
huBERT 32K ~4.60 ~1.65 WordPiece
PULI-GPT-3SX 50K ~4.45 ~1.71 BPE
PULI-GPTrio 150K ~4.65 ~1.62 BPE (Trilingual)
GPT-4o 200K 4.2–4.4 1.6–1.8 BPE
Llama 3.1 128K ~4.10 ~1.85 BPE

Runs of emese-tech emese-tokenizer-32k on huggingface.co

0
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs

More Information About emese-tokenizer-32k huggingface.co Model

More emese-tokenizer-32k license Visit here:

https://choosealicense.com/licenses/mit

emese-tokenizer-32k huggingface.co

emese-tokenizer-32k huggingface.co is an AI model on huggingface.co that provides emese-tokenizer-32k's model effect (), which can be used instantly with this emese-tech emese-tokenizer-32k model. huggingface.co supports a free trial of the emese-tokenizer-32k model, and also provides paid use of the emese-tokenizer-32k. Support call emese-tokenizer-32k model through api, including Node.js, Python, http.

emese-tokenizer-32k huggingface.co Url

https://huggingface.co/emese-tech/emese-tokenizer-32k

emese-tech emese-tokenizer-32k online free

emese-tokenizer-32k huggingface.co is an online trial and call api platform, which integrates emese-tokenizer-32k's modeling effects, including api services, and provides a free online trial of emese-tokenizer-32k, you can try emese-tokenizer-32k online for free by clicking the link below.

emese-tech emese-tokenizer-32k online free url in huggingface.co:

https://huggingface.co/emese-tech/emese-tokenizer-32k

emese-tokenizer-32k install

emese-tokenizer-32k is an open source model from GitHub that offers a free installation service, and any user can find emese-tokenizer-32k on GitHub to install. At the same time, huggingface.co provides the effect of emese-tokenizer-32k install, users can directly use emese-tokenizer-32k installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

emese-tokenizer-32k install url in huggingface.co:

https://huggingface.co/emese-tech/emese-tokenizer-32k

Url of emese-tokenizer-32k

emese-tokenizer-32k huggingface.co Url

Provider of emese-tokenizer-32k huggingface.co

emese-tech
ORGANIZATIONS
huggingface.co

Other API from emese-tech

 huggingface.co
emese-tech/csermely

Total runs: 824
Run Growth: 813
Growth Rate: 98.91%
Updated:September 08 2026
huggingface.co
emese-tech/patak

Total runs: 393
Run Growth: 393
Growth Rate: 100.00%
Updated:September 07 2026
huggingface.co
emese-tech/er

Total runs: 363
Run Growth: 363
Growth Rate: 100.00%
Updated:September 07 2026
huggingface.co
emese-tech/folyo

Total runs: 342
Run Growth: 342
Growth Rate: 100.00%
Updated:September 07 2026
huggingface.co
emese-tech/patak-mlx

Total runs: 202
Run Growth: 202
Growth Rate: 100.00%
Updated:September 07 2026
huggingface.co
emese-tech/er-mlx

Total runs: 194
Run Growth: 194
Growth Rate: 100.00%
Updated:September 07 2026
huggingface.co
emese-tech/folyo-mlx

Total runs: 154
Run Growth: 154
Growth Rate: 100.00%
Updated:September 07 2026
huggingface.co
emese-tech/folyo-gguf

Total runs: 100
Run Growth: 100
Growth Rate: 100.00%
Updated:September 07 2026
huggingface.co
emese-tech/patak-gguf

Total runs: 85
Run Growth: 85
Growth Rate: 100.00%
Updated:September 07 2026