csebe / modernbert_bioinfo_voc

ModernBert with injection of vocabulary

ModernBert with injected tools Bioinformatics vocabulary.

The vocabulary added in the pretrained language model is a list of bioinformatics tool names and binary commands extracted from different Bioinformatics tools Knowledges Bases :

• Bio.tools

J. Ison, K. Rapacki, H. Ménager, M. Kalas, E. Rydza, P. Chmura, C. Anthon, N. Beard, K. Berka, D. Bolser, T. Booth, A. Bretaudeau, J. Brezovsky, R. Casadio, G. Cesareni, F. Coppens, M. Cornell, G. Cuccuru, and et al. Tools and data services registry: a community effort to document bioinformatics resources. Nucleic Acids Research,44(D1):D38–D47, Jan. 2016. ISSN 0305-1048. doi: 10.1093/nar/gkv1116. URL https://doi.org/10.1093/nar/gkv1116 .

• Bioconda

B. Grüning, R. Dale, A. Sjödin, B. A. Chapman, J. Rowe, C. H. Tomkins-Tinch, R. Valieris, and J. Köster. Bioconda: sustainable and comprehensive software distribution for the life sciences. Nature Methods, 15 (7):475–476, July 2018. ISSN 1548-7105. doi: 10.1038/s41592-018-0046-7. URL https://www.nature.com/articles/s41592-018-0046-7 .

• Biocontainers

F. da Veiga Leprevost, B. A. Grüning, S. Alves Aflitos, H. L. Röst, J. Uszkoreit, H. Barsnes, M. Vaudel, and et al. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics, 33 (16):2580–2582, Aug. 2017. ISSN 1367-4803. doi: 10.1093/bioinformatics/btx192. URL https://doi.org/10.1093/bioinformatics/btx192 .

• and Bioweb (Knowledge base manually curated at Institut Pasteur, France)

Vocabulary Injection Strategy

• Tokenizer Update : new words are added to the existing tokenizer of ModernBert

• Embedding Initialization : embeddings for newly introduced words are initialized as the average of the embeddings of their constituent existing words

Model Adaptation

After extending the vocabulary, two main strategies can be applied when performing NER with this model:

• Option 1: Fine-Tuning (Recommended) : the language model is fine-tuned on domain-specific corpora to better integrate the new vocabulary into its internal representations

• Option 2: No Fine-Tuning : the extended model is used directly without further training

Use in

This model is used to do NER experiments on (CPL-Corpus)[] and (CPL-Code)[].

This model can be used with :

• https://github.com/ClemenceS/nlstruct
• Other experiments are presented in upcoming