UMCU / CardioDeBERTa.nl

Model Card

This is DeBERTaV2 pre-trained language model from scratch on the DutchMedicalText corpus. A clinical model, continously-pretrained on UMCU clinical texts will uploaded later and linked here.

Model Details

About 410M parameters, with a 1024 token context length

Model Description

• Developed by: Bram van Es - at UPOD UMCU / MEDxAI
• Funded by : UMCU / Google TPU
• Model type: DeBERTaV2
• Language(s) (NLP): Dutch
• License: GPL-3

Model Sources [optional]

• Repository: code for training
• Paper: coming

Intended use

This model is directly suitable for Masked Language Modeling and can be finetuned for token/sequence classification, contrastive embeddings, relationship extraction and other downstream tasks.

Bias, Risks, and Limitations

This model was not filtered for bias. As for any language model, do not blindly accept the generated output. This is not a causal model, and it is not finetuned in any away for clinical decision support tasks.

Training Details

Training Data

Trained on about 80GB of Dutch medical texts, ranging van guidelines to patient cases reports.

Training Procedure

Preprocessing

• Deidentification with DEDUCE
• Removal of repetitive phrases and repetitive non-word characters
• Cleaning with FTFY
• Chunking of each document based on token counts. We do not

Training Hyperparameters

• Training regime: bf16 mixed precision
• learning rate: 1e-4 - 2e-4
• number of warmup steps: 5000
• steps per epoch: 50000
• weight decacy: 0.001

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019) .

• Hardware Type: TPUv4
• Hours used: 400+
• Cloud Provider: Google
• Compute Region: US-WEST2 en EUROPE-WEST4
• Carbon Emitted: 100kg+, compensated