UMCU / ICD10_classifier_base_English

We finetuned the SapBERT encoder model on the synthetic ICD10 set by FiscaAI .

We trimmed the ICD10 codes to X## , i.e. we trimmed the dot, and then kept only the ICD10 codes that had more than 1000 occurrences, after which we kept the texts that had at least one of the remaining ICD10 codes, this resulted in close to 950 ICD10 codes uses as labels for the finetuning with about 700.000 samples.

We use multilabel finetuning with a head of 3 dense layers, with 10% dropout and GELU activation.

We used class weighting with log smoothing.

For the

TLDR;

• ICD10 trimmed after dot, A10.42 -> A10
• minimum_token_count 16
• minimum_label_count 1000
• sequence_length 256
• log-smoothed class weights
• 3 dense layers, 10% dropout, GELU
• 5 epochs
• batch size 16
• max learning rate 2e-5
• repeated linear warmup (10.000 steps) + cosine decay (50.000 steps)

When you load this model for feature extraction or classification use

from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
    "UMCU/ICD10_classifier_base_English",
    trust_remote_code=True,
)

or

from transformers import pipeline

le_pipe = pipeline('text-classification',trust_remote_code=True)

The macro f1 score is 0.65. We suggest to use it primarily for feature extraction or for use as an embedder.