Total runs:
24.1K
Run Growth:
13.5K
Growth Rate:
56.09%
Sponsored by
Vmake Labs Video Enhancer
ytu-ce-cosmos / modernbert-tr-base-1k
Model's Last Updated: July 10 2026
fill-mask
Introduction of modernbert-tr-base-1k
Model Details of modernbert-tr-base-1k
ModernBERT-TR
A Modern Encoder Foundation Model for Turkish
Besher Alkurdi, Himmet Toprak Kesgin, Muzaffer Kaan Yuce, Mehmet Fatih Amasyali
Overview
ModernBERT-TR is a 150M-parameter Turkish encoder pretrained from scratch on 144.4B tokens using the ModernBERT architecture. It uses a custom 50K WordPiece tokenizer optimized for Turkish morphology.
Architecture: 22 layers, 768 hidden, 12 heads, RoPE, GLU, alternating local-global attention, Flash Attention, sequence-packed training.
Results
Frozen Linear Probing (11 Turkish NLP tasks)
| Model | Params | Avg |
|---|---|---|
| ModernBERT-TR (ours) | 150M | 60.2 |
| Turkish-E5-large | 560M | 53.2 |
| mmBERT | 307M | 54.9 |
| TabiBERT | ~150M | 49.1 |
| BERTurk | 111M | 35.3 |
+13.1% relative over next-best. +70.3% relative over BERTurk. Outperforms models up to 4x larger.
TabiBench Full Fine-Tuning (28 tasks)
| Model | Params | Avg |
|---|---|---|
| ModernBERT-TR (ours) | 150M | 77.28 |
| TabiBERT | ~150M | 77.58 |
| BERTurk | 110M | 75.96 |
Leads in 5/8 categories (text classification, STS, NLI, academic understanding, information retrieval). TabiBERT leads in code retrieval and QA (trained on code/math data).
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("ytu-ce-cosmos/modernbert-tr-base-1k")
model = AutoModelForMaskedLM.from_pretrained("ytu-ce-cosmos/modernbert-tr-base-1k")
text = "Türkiye'nin başkenti [MASK]'dır."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
Training Details
| Data | FineWeb-2 Turkish (41.2B tokens) + BertTurk Corpus 5x (31.0B tokens) = 72.2B/epoch, 2 epochs |
| Tokenizer | 50K WordPiece, trained on Turkish data |
| Optimizer | StableAdamW, peak LR 2e-4, cosine schedule |
| Batch size | 256 sequences (262K tokens/step) |
| MLM masking | 30% (train) / 15% (eval) |
| Hardware | 4x NVIDIA H100, 623 GPU-hours |
| Precision | BF16 mixed precision |
| Context | 1,024 tokens |
Citation
@article{alkurdi2025modernberttr,
title={ModernBERT-TR: A Modern Encoder Foundation Model for Turkish},
author={Alkurdi, Besher and Kesgin, Himmet Toprak and Yuce, Muzaffer Kaan and Amasyali, Mehmet Fatih},
year={2025}
}
Acknowledgments
Supported by Yildiz Technical University (FDK-2024-6070) and TUBITAK (124E055). Built on the ModernBERT codebase with FineWeb-2 and BertTurk Corpus data.
Runs of ytu-ce-cosmos modernbert-tr-base-1k on huggingface.co
1.7K
Total runs
0
24-hour runs
-368
3-day runs
-583
7-day runs
-1.3K
30-day runs
More Information About modernbert-tr-base-1k huggingface.co Model
Url of modernbert-tr-base-1k
modernbert-tr-base-1k huggingface.co Url
Provider of modernbert-tr-base-1k huggingface.co
Other API from ytu-ce-cosmos
Updated:December 01 2025
Total runs:
18.6K
Run Growth:
115
Growth Rate:
0.62%
Total runs:
3.1K
Run Growth:
-363
Growth Rate:
-11.62%
Total runs:
2.0K
Run Growth:
-9.2K
Growth Rate:
-468.19%
Total runs:
1.1K
Run Growth:
672
Growth Rate:
61.54%
Total runs:
1.1K
Run Growth:
-546
Growth Rate:
-50.79%
Total runs:
1.0K
Run Growth:
934
Growth Rate:
92.84%
Total runs:
987
Run Growth:
-76
Growth Rate:
-7.70%
Total runs:
854
Run Growth:
260
Growth Rate:
30.44%
Total runs:
723
Run Growth:
-55
Growth Rate:
-7.61%
Total runs:
701
Run Growth:
-380
Growth Rate:
-54.21%
Total runs:
496
Run Growth:
-524
Growth Rate:
-105.65%
Total runs:
471
Run Growth:
-242
Growth Rate:
-51.38%
Total runs:
437
Run Growth:
-69
Growth Rate:
-15.79%
Total runs:
303
Run Growth:
37
Growth Rate:
12.21%
Total runs:
281
Run Growth:
251
Growth Rate:
89.32%
Total runs:
243
Run Growth:
-196
Growth Rate:
-80.66%
Total runs:
202
Run Growth:
27
Growth Rate:
13.37%
Total runs:
199
Run Growth:
163
Growth Rate:
81.91%
Total runs:
196
Run Growth:
-28
Growth Rate:
-14.29%
Total runs:
186
Run Growth:
29
Growth Rate:
15.59%
Total runs:
181
Run Growth:
-33
Growth Rate:
-18.23%
Total runs:
134
Run Growth:
41
Growth Rate:
30.60%
Total runs:
123
Run Growth:
67
Growth Rate:
54.47%
Total runs:
100
Run Growth:
100
Growth Rate:
100.00%
Total runs:
93
Run Growth:
23
Growth Rate:
24.73%
Total runs:
91
Run Growth:
38
Growth Rate:
41.76%
Total runs:
85
Run Growth:
26
Growth Rate:
30.59%
Total runs:
82
Run Growth:
47
Growth Rate:
57.32%
Total runs:
55
Run Growth:
34
Growth Rate:
61.82%
Total runs:
47
Run Growth:
2
Growth Rate:
4.26%
Total runs:
35
Run Growth:
-465
Growth Rate:
-1328.57%
Total runs:
34
Run Growth:
22
Growth Rate:
64.71%
Total runs:
33
Run Growth:
33
Growth Rate:
100.00%
Total runs:
32
Run Growth:
29
Growth Rate:
90.63%
Total runs:
28
Run Growth:
6
Growth Rate:
21.43%
Total runs:
26
Run Growth:
26
Growth Rate:
100.00%
Total runs:
25
Run Growth:
-125
Growth Rate:
-500.00%
Total runs:
25
Run Growth:
-51
Growth Rate:
-204.00%
Total runs:
24
Run Growth:
18
Growth Rate:
75.00%
Total runs:
24
Run Growth:
24
Growth Rate:
100.00%
Total runs:
21
Run Growth:
21
Growth Rate:
100.00%
Total runs:
19
Run Growth:
-31
Growth Rate:
-163.16%
Total runs:
16
Run Growth:
6
Growth Rate:
37.50%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%