Total runs:
989.9K
Run Growth:
22.3K
Growth Rate:
2.25%
Sponsored by
PoYo.AI
almanach / ModernCamemBERT-bio-v2-large
Model's Last Updated: August 03 2026
fill-mask
Introduction of ModernCamemBERT-bio-v2-large
Model Details of ModernCamemBERT-bio-v2-large
ModernCamemBERT-bio-v2-large
ModernCamemBERT-bio-v2-large is a French biomedical encoder for document-level tasks. It is obtained by continuing the pretraining of ModernCamemBERT-bio-large with an OntoBook phase, with an 8,192-token context window. On our French biomedical document benchmark it reaches 72.8 micro-F1, the best score we measured for coding and long-document classification at this size.
Authors
Rian Touchent and Eric de la Clergerie
Sorbonne Université, Inria Paris
Overview
ModernCamemBERT-bio is already adapted to French biomedical text through a CLM detour. The v2 variant adds a second continued-pretraining phase that we call OntoBook. Rather than plain masked language modeling, we train on ontology-grounded synthetic textbooks built from CIM-10, CCAM and ATC ontology walks, mixed with clinical text. The goal is to inject structured coding knowledge without adding any task-specific head. This model is our document specialist.
| Architecture | ModernBERT |
|---|---|
| Parameters | 350M |
| Context length | 8,192 tokens |
| Language | French |
| Base model | almanach/ModernCamemBERT-bio-large |
Usage
from transformers import AutoTokenizer, AutoModelForMaskedLM
model_id = "rntc/ModernCamemBERT-bio-v2-large"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)
text = "Le patient présente une [MASK] aiguë du myocarde."
inputs = tokenizer(text, return_tensors="pt")
logits = model(**inputs).logits
i = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
print(tokenizer.decode(logits[0, i].argmax(-1)))
Evaluation
Per-task results on the document core of our French biomedical benchmark, scored with micro-F1 and averaged over nine seeds.
| Model | FrACCO-30 | FrACCO-100 | CANTEMIST | MORFITT | MedDialog-FR | Avg |
|---|---|---|---|---|---|---|
| ModernCamemBERT-bio-v2-large | 82.0 | 67.8 | 74.6 | 74.3 | 65.2 | 72.8 |
| ModernCamemBERT-bio-v2-base | 78.0 | 63.7 | 70.9 | 73.5 | 63.7 | 70.0 |
| ModernCamemBERT-bio-base | 74.2 | 61.1 | 71.2 | 72.9 | 63.7 | 68.6 |
| DoctoModernBERT | 74.0 | 58.3 | 69.3 | 73.9 | 64.7 | 68.0 |
| DrBERT | 54.2 | 39.6 | 41.2 | 71.0 | 64.1 | 54.0 |
| DoctoBERT | 51.6 | 34.7 | 36.2 | 73.5 | 62.1 | 51.6 |
| CamemBERT-bio | 44.0 | 22.3 | 17.7 | 70.9 | 45.8 | 40.1 |
ModernCamemBERT-bio-v2-large has the best average and the best score on each coding task. It scores 4.2 points above ModernCamemBERT-bio-base, the strongest publicly available French biomedical encoder we evaluated.
Environmental impact
The OntoBook phase reported here ran for about 3.4 GPU-hours on a single H100, on the Jean Zay cluster (GENCI-IDRIS) in France. We estimate roughly 0.10 kg CO2eq for this phase. This figure covers the continued-pretraining step only and does not include the pretraining of the base model.
License
MIT
Citation
@inproceedings{touchent:hal-05697506,
TITLE = {{OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining}},
AUTHOR = {Touchent, Rian and de la Clergerie, {\'E}ric},
URL = {https://hal.science/hal-05697506},
BOOKTITLE = {{Proceedings of Knowledge Graphs and Large Language Models Workshop}},
ADDRESS = {Palma de Mallorca, Spain},
YEAR = {2026},
MONTH = May,
PDF = {https://hal.science/hal-05697506v1/file/main.pdf},
HAL_ID = {hal-05697506},
HAL_VERSION = {v1},
}
@misc{touchent2026causallanguagemodelingdetour,
title={A Causal Language Modeling Detour Improves Encoder Continued Pretraining},
author={Rian Touchent and Eric de la Clergerie},
year={2026},
eprint={2605.12438},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2605.12438},
}
Runs of almanach ModernCamemBERT-bio-v2-large on huggingface.co
34
Total runs
17
24-hour runs
20
3-day runs
22
7-day runs
-93
30-day runs
More Information About ModernCamemBERT-bio-v2-large huggingface.co Model
Url of ModernCamemBERT-bio-v2-large
ModernCamemBERT-bio-v2-large huggingface.co Url
Provider of ModernCamemBERT-bio-v2-large huggingface.co
Other API from almanach
Updated:June 23 2025
Total runs:
45.5K
Run Growth:
41.2K
Growth Rate:
85.52%
Total runs:
5.8K
Run Growth:
402
Growth Rate:
7.09%
Total runs:
2.9K
Run Growth:
-256
Growth Rate:
-8.50%
Total runs:
2.2K
Run Growth:
-1.3K
Growth Rate:
-51.09%
Total runs:
1.3K
Run Growth:
1.3K
Growth Rate:
100.00%
Total runs:
926
Run Growth:
703
Growth Rate:
76.16%
Total runs:
730
Run Growth:
526
Growth Rate:
72.75%
Total runs:
496
Run Growth:
404
Growth Rate:
80.48%
Total runs:
484
Run Growth:
-117
Growth Rate:
-26.65%
Total runs:
380
Run Growth:
-54
Growth Rate:
-15.65%
Total runs:
298
Run Growth:
82
Growth Rate:
29.29%
Total runs:
285
Run Growth:
174
Growth Rate:
62.14%
Total runs:
269
Run Growth:
247
Growth Rate:
91.82%
Total runs:
250
Run Growth:
-24
Growth Rate:
-9.64%
Total runs:
229
Run Growth:
229
Growth Rate:
100.00%
Total runs:
228
Run Growth:
228
Growth Rate:
100.00%
Total runs:
173
Run Growth:
110
Growth Rate:
63.58%
Total runs:
169
Run Growth:
169
Growth Rate:
100.00%
Total runs:
162
Run Growth:
132
Growth Rate:
84.08%
Total runs:
141
Run Growth:
141
Growth Rate:
100.00%
Total runs:
105
Run Growth:
12
Growth Rate:
11.65%
Total runs:
92
Run Growth:
-319
Growth Rate:
-350.55%
Total runs:
87
Run Growth:
67
Growth Rate:
80.72%
Total runs:
81
Run Growth:
56
Growth Rate:
73.68%
Total runs:
69
Run Growth:
50
Growth Rate:
72.46%
Total runs:
59
Run Growth:
27
Growth Rate:
45.76%
Total runs:
58
Run Growth:
29
Growth Rate:
43.94%
Total runs:
40
Run Growth:
-330
Growth Rate:
-846.15%
Total runs:
35
Run Growth:
72
Growth Rate:
100.00%
Total runs:
34
Run Growth:
26
Growth Rate:
78.79%
Total runs:
23
Run Growth:
-64
Growth Rate:
-278.26%
Total runs:
19
Run Growth:
1
Growth Rate:
5.56%
Total runs:
18
Run Growth:
9
Growth Rate:
52.94%
Total runs:
16
Run Growth:
-6
Growth Rate:
-37.50%
Total runs:
15
Run Growth:
9
Growth Rate:
60.00%
Total runs:
14
Run Growth:
9
Growth Rate:
60.00%
Total runs:
14
Run Growth:
-70
Growth Rate:
-500.00%
Total runs:
13
Run Growth:
-2
Growth Rate:
-15.38%
Total runs:
13
Run Growth:
-9
Growth Rate:
-69.23%
Total runs:
13
Run Growth:
-6
Growth Rate:
-54.55%
Total runs:
13
Run Growth:
-5
Growth Rate:
-38.46%
Total runs:
13
Run Growth:
3
Growth Rate:
23.08%
Total runs:
12
Run Growth:
4
Growth Rate:
30.77%
Total runs:
11
Run Growth:
-5
Growth Rate:
-45.45%
Total runs:
11
Run Growth:
5
Growth Rate:
41.67%
Total runs:
10
Run Growth:
-5
Growth Rate:
-50.00%
Total runs:
10
Run Growth:
-5
Growth Rate:
-50.00%
Total runs:
10
Run Growth:
4
Growth Rate:
40.00%
Total runs:
9
Run Growth:
9
Growth Rate:
100.00%
Total runs:
9
Run Growth:
-60
Growth Rate:
-146.34%
Total runs:
9
Run Growth:
-5
Growth Rate:
-55.56%
Total runs:
9
Run Growth:
-5
Growth Rate:
-55.56%
Total runs:
8
Run Growth:
-8
Growth Rate:
-100.00%
Total runs:
8
Run Growth:
-9
Growth Rate:
-112.50%
Total runs:
7
Run Growth:
7
Growth Rate:
100.00%
Total runs:
6
Run Growth:
-9
Growth Rate:
-150.00%
Total runs:
4
Run Growth:
-6
Growth Rate:
-150.00%
Total runs:
4
Run Growth:
-6
Growth Rate:
-150.00%
Total runs:
4
Run Growth:
-12
Growth Rate:
-300.00%
Total runs:
4
Run Growth:
2
Growth Rate:
50.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%