Total runs:
959.7K
Run Growth:
-12.6K
Growth Rate:
-1.31%
Sponsored by
APIMart
almanach / ModernCamemBERT-bio-v2-base
Model's Last Updated: August 03 2026
fill-mask
Introduction of ModernCamemBERT-bio-v2-base
Model Details of ModernCamemBERT-bio-v2-base
ModernCamemBERT-bio-v2-base
ModernCamemBERT-bio-v2-base is a French biomedical encoder for document-level tasks. It is obtained by continuing the pretraining of ModernCamemBERT-bio-base with an OntoBook phase, with an 8,192-token context window. On our French biomedical document benchmark it reaches 70.0 micro-F1 at 150M parameters. A larger version is available as ModernCamemBERT-bio-v2-large .
Authors
Rian Touchent and Eric de la Clergerie
Sorbonne Université, Inria Paris
Overview
ModernCamemBERT-bio is already adapted to French biomedical text through a CLM detour. The v2 variant adds a second continued-pretraining phase that we call OntoBook. Instead of plain masked language modeling, we train on ontology-grounded synthetic textbooks built from CIM-10, CCAM and ATC ontology walks, mixed with clinical text. The aim is to inject structured coding knowledge without adding a task-specific head. This is the base-size document specialist.
| Architecture | ModernBERT |
|---|---|
| Parameters | 150M |
| Context length | 8,192 tokens |
| Language | French |
| Base model | almanach/ModernCamemBERT-bio-base |
Usage
from transformers import AutoTokenizer, AutoModelForMaskedLM
model_id = "rntc/ModernCamemBERT-bio-v2-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)
text = "Le patient présente une [MASK] aiguë du myocarde."
inputs = tokenizer(text, return_tensors="pt")
logits = model(**inputs).logits
i = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
print(tokenizer.decode(logits[0, i].argmax(-1)))
Evaluation
Per-task results on the document core of our French biomedical benchmark, scored with micro-F1 and averaged over nine seeds.
| Model | FrACCO-30 | FrACCO-100 | CANTEMIST | MORFITT | MedDialog-FR | Avg |
|---|---|---|---|---|---|---|
| ModernCamemBERT-bio-v2-large | 82.0 | 67.8 | 74.6 | 74.3 | 65.2 | 72.8 |
| ModernCamemBERT-bio-v2-base | 78.0 | 63.7 | 70.9 | 73.5 | 63.7 | 70.0 |
| ModernCamemBERT-bio-base | 74.2 | 61.1 | 71.2 | 72.9 | 63.7 | 68.6 |
| DoctoModernBERT | 74.0 | 58.3 | 69.3 | 73.9 | 64.7 | 68.0 |
| DrBERT | 54.2 | 39.6 | 41.2 | 71.0 | 64.1 | 54.0 |
| DoctoBERT | 51.6 | 34.7 | 36.2 | 73.5 | 62.1 | 51.6 |
| CamemBERT-bio | 44.0 | 22.3 | 17.7 | 70.9 | 45.8 | 40.1 |
At base size, ModernCamemBERT-bio-v2-base leads the average and scores 1.4 points above its ModernCamemBERT-bio-base starting point. ModernCamemBERT-bio-v2-large scores 2.8 points higher at 350M parameters.
Environmental impact
The OntoBook phase reported here ran for about 1.3 GPU-hours on a single H100, on the Jean Zay cluster (GENCI-IDRIS) in France. We estimate roughly 0.04 kg CO2eq for this phase. This figure covers the continued-pretraining step only and does not include the pretraining of the base model.
License
MIT
Citation
@inproceedings{touchent:hal-05697506,
TITLE = {{OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining}},
AUTHOR = {Touchent, Rian and de la Clergerie, {\'E}ric},
URL = {https://hal.science/hal-05697506},
BOOKTITLE = {{Proceedings of Knowledge Graphs and Large Language Models Workshop}},
ADDRESS = {Palma de Mallorca, Spain},
YEAR = {2026},
MONTH = May,
PDF = {https://hal.science/hal-05697506v1/file/main.pdf},
HAL_ID = {hal-05697506},
HAL_VERSION = {v1},
}
@misc{touchent2026causallanguagemodelingdetour,
title={A Causal Language Modeling Detour Improves Encoder Continued Pretraining},
author={Rian Touchent and Eric de la Clergerie},
year={2026},
eprint={2605.12438},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2605.12438},
}
Runs of almanach ModernCamemBERT-bio-v2-base on huggingface.co
19
Total runs
0
24-hour runs
2
3-day runs
5
7-day runs
-68
30-day runs
More Information About ModernCamemBERT-bio-v2-base huggingface.co Model
Url of ModernCamemBERT-bio-v2-base
ModernCamemBERT-bio-v2-base huggingface.co Url
Provider of ModernCamemBERT-bio-v2-base huggingface.co
Other API from almanach
Updated:June 23 2025
Total runs:
50.3K
Run Growth:
43.3K
Growth Rate:
86.06%
Total runs:
5.4K
Run Growth:
-521
Growth Rate:
-9.58%
Total runs:
3.4K
Run Growth:
467
Growth Rate:
13.63%
Total runs:
3.1K
Run Growth:
-2.5K
Growth Rate:
-80.02%
Total runs:
2.4K
Run Growth:
-1.3K
Growth Rate:
-56.69%
Total runs:
1.2K
Run Growth:
1.2K
Growth Rate:
100.00%
Total runs:
917
Run Growth:
669
Growth Rate:
72.96%
Total runs:
710
Run Growth:
558
Growth Rate:
78.59%
Total runs:
504
Run Growth:
356
Growth Rate:
70.63%
Total runs:
460
Run Growth:
-60
Growth Rate:
-13.04%
Total runs:
316
Run Growth:
164
Growth Rate:
51.90%
Total runs:
297
Run Growth:
-127
Growth Rate:
-42.76%
Total runs:
268
Run Growth:
246
Growth Rate:
91.79%
Total runs:
261
Run Growth:
100
Growth Rate:
38.31%
Total runs:
229
Run Growth:
229
Growth Rate:
100.00%
Total runs:
228
Run Growth:
-41
Growth Rate:
-17.98%
Total runs:
228
Run Growth:
228
Growth Rate:
100.00%
Total runs:
169
Run Growth:
169
Growth Rate:
100.00%
Total runs:
164
Run Growth:
121
Growth Rate:
73.78%
Total runs:
140
Run Growth:
136
Growth Rate:
97.14%
Total runs:
135
Run Growth:
111
Growth Rate:
82.22%
Total runs:
117
Run Growth:
28
Growth Rate:
23.93%
Total runs:
78
Run Growth:
-326
Growth Rate:
-417.95%
Total runs:
74
Run Growth:
62
Growth Rate:
83.78%
Total runs:
72
Run Growth:
72
Growth Rate:
100.00%
Total runs:
71
Run Growth:
41
Growth Rate:
57.75%
Total runs:
68
Run Growth:
51
Growth Rate:
75.00%
Total runs:
66
Run Growth:
51
Growth Rate:
77.27%
Total runs:
56
Run Growth:
46
Growth Rate:
82.14%
Total runs:
41
Run Growth:
-60
Growth Rate:
-146.34%
Total runs:
34
Run Growth:
-334
Growth Rate:
-982.35%
Total runs:
31
Run Growth:
24
Growth Rate:
77.42%
Total runs:
16
Run Growth:
-1
Growth Rate:
-6.25%
Total runs:
16
Run Growth:
-2
Growth Rate:
-12.50%
Total runs:
14
Run Growth:
5
Growth Rate:
35.71%
Total runs:
13
Run Growth:
4
Growth Rate:
30.77%
Total runs:
13
Run Growth:
-113
Growth Rate:
-869.23%
Total runs:
13
Run Growth:
7
Growth Rate:
53.85%
Total runs:
12
Run Growth:
-73
Growth Rate:
-608.33%
Total runs:
12
Run Growth:
-1
Growth Rate:
-8.33%
Total runs:
12
Run Growth:
2
Growth Rate:
16.67%
Total runs:
11
Run Growth:
-9
Growth Rate:
-81.82%
Total runs:
10
Run Growth:
-7
Growth Rate:
-70.00%
Total runs:
10
Run Growth:
4
Growth Rate:
40.00%
Total runs:
9
Run Growth:
9
Growth Rate:
100.00%
Total runs:
9
Run Growth:
-4
Growth Rate:
-44.44%
Total runs:
9
Run Growth:
-5
Growth Rate:
-55.56%
Total runs:
7
Run Growth:
-7
Growth Rate:
-100.00%
Total runs:
7
Run Growth:
-7
Growth Rate:
-100.00%
Total runs:
7
Run Growth:
-6
Growth Rate:
-85.71%
Total runs:
7
Run Growth:
-7
Growth Rate:
-100.00%
Total runs:
7
Run Growth:
7
Growth Rate:
100.00%
Total runs:
6
Run Growth:
-10
Growth Rate:
-166.67%
Total runs:
6
Run Growth:
-7
Growth Rate:
-116.67%
Total runs:
6
Run Growth:
2
Growth Rate:
33.33%
Total runs:
5
Run Growth:
-9
Growth Rate:
-180.00%
Total runs:
4
Run Growth:
-5
Growth Rate:
-125.00%
Total runs:
4
Run Growth:
-4
Growth Rate:
-100.00%
Total runs:
4
Run Growth:
-29
Growth Rate:
-725.00%
Total runs:
4
Run Growth:
-9
Growth Rate:
-225.00%
Total runs:
2
Run Growth:
-14
Growth Rate:
-700.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%