StanfordShahLab / clmbr-t-base

CLMBR-T-Base

This is a 141 million parameter autoregressive foundation model pretrained on 2.57 million deidentified EHRs from Stanford Medicine.

This is the model from (Wornow et al. 2023) , and is based on the CLMBR architecture originally described in (Steinberg et al. 2021)

As input, this model expects a sequence of coded medical events that have been mapped to Standard Concepts within the OMOP-CDM vocabulary . The model generates representations of patients which can then be used for downstream prediction tasks.

Input patients should be provided in the MEDS schema.

Model Details

Model Description

• Developed by: Shah lab @ Stanford University
• Funded by: Stanford Healthcare
• Shared by: Shah lab @ Stanford University
• Model type: CLMBR (Steinberg et al. 2021)
• Language(s) (NLP): Electronic health record codes
• License: CC-BY NC 4.0
• Finetuned from model: N/A -- trained from scratch

Model Sources

• Website: https://ehrshot.stanford.edu/
• Gitub: https://github.com/som-shahlab/ehrshot-benchmark/
• Paper: EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models

Uses

This model is intended to generate representations for patients based on the structured data within their electronic health record. These representations can then be used for downstream tasks such as predicting diagnoses, detecting anomalies, or doing propensity score matching for causal inference.

Direct Use

You will likely want to tune the model for your downstream use case.

Out-of-Scope Use

This model is for research purposes only. It is not for use in any real-world decision making that impacts patients, providers, or hospital operations.

Bias, Risks, and Limitations

This model was trained on a corpus of 2.57 million patients from Stanford Medicine. The model will thus reflect the patterns of how care is delivered at Stanford Medicine, in addition to the racial and socioeconomic makeup of Stanford Medicine's patient base. This model may not generalize well to other hospitals and demographic mixes.

While this is technically a generative model, we have not tested its generative abilities and thus do not anticipate it being used to generate synthetic EHR records. We aim to explore its generative abilities in future work.

How to Get Started with the Model

Use the code below to get started with the model.

First, download the necessary libraries.

pip install torch==2.1.1 femr==0.2.3 datasets==2.15.0 xformers transformers==4.35.2

Second, run the following Python script to run inference on a single patient:

import femr.models.transformer
import torch
import femr.models.tokenizer
import femr.models.processor
import datetime

model_name = "StanfordShahLab/clmbr-t-base"

# Load tokenizer / batch loader
tokenizer = femr.models.tokenizer.FEMRTokenizer.from_pretrained(model_name)
batch_processor = femr.models.processor.FEMRBatchProcessor(tokenizer)

# Load model
model = femr.models.transformer.FEMRModel.from_pretrained(model_name)

# Create an example patient to run inference on
# This patient follows the MEDS schema: https://github.com/Medical-Event-Data-Standard
example_patient = {
    'patient_id': 30,
    'events': [{
        'time': datetime.datetime(2011, 5, 8),
        'measurements': [
            {'code': 'SNOMED/184099003'},
            {'code': 'Visit/IP'},
        ],
    },
    {
        'time': datetime.datetime(2012, 6, 9),
        'measurements': [
            {'code': 'Visit/OP'},
            {'code': 'SNOMED/3950001'}
        ],
    }]
}

raw_batch = batch_processor.convert_patient(example_patient, tensor_type="pt")
batch = batch_processor.collate([raw_batch])

# Run model
with torch.no_grad():
    _, result = model(**batch)
    print(result['timestamps'].cpu().numpy().astype('datetime64[s]'))
    print(result['patient_ids'])
    print(result['representations'])

Training Details

Full training details are provided in our accompanying paper, EHRSHOT (Wornow et al. 2023) .

Training Data

The model is trained on 2.57 million patients from the Stanford Medicine Research Data Repository (STARR) , which contains EHR data from both Stanford Health Care (primarily adult care) and Lucile Packard Children’s Hospital (primarily pediatric care). The dataset contains only structured data (i.e. no clinical text or images) and covers demographics (e.g. age, sex, race), diagnoses, procedures, laboratory results, medication prescriptions, and other coded clinical observations. The data is formatted according to the Observational Medical Outcomes Partnership Common Data Model (OMOP-CDM) . All data that we work with is deidentified.

Training Procedure

We train our model using an autoregressive next code prediction objective, i.e. predict the next code in a patient's timeline given their previous codes.

Preprocessing

We use the FEMR Python library for data preprocessing.

Training Hyperparameters

• Learning rate: 0.00001
• Context window size: 496
• Internal dropout: 0
• Layers: 12
• Hidden dimension: 768

Evaluation

We evaluate this model on the EHRSHOT benchmark .

Information on this benchmark, tasks, and results are detailed in Wornow et al. 2023

Technical Specifications

This model uses the CLMBR architecture from (Steinberg et al. 2021) . The objective is an autoregressive next token prediction task. Please see Wornow et al. 2023 for more details on the specific model architecture.

Vocabulary

CLMBR is a language model and requires defining a token vocabulary V . However, unlike natural languages, the vocabulary of a structured EHR language model is defined by medical codes . Here tokens map to standardized concepts in medical ontologies. Since the union of all tokens from all ontologies, V_all , results in a prohibitively large vocabuary, we derive ~V by filtering to the top k most frequent codes as follows:

1. Knowledge Graphs (G): A set of n medical ontologies (knowledge graphs), G = ({G_1, G_2, ..., G_n}) , defined by Athena's OMOP Vocabulary List .
2. Medical Codes as Tokens: Each knowledge graph G_i has a set of unique medical codes M_i . The union of all these codes serve as the tokens in our complete vocabulary V_all = M_1 ∪ M_2 ∪ ... ∪ M_n . Our final, filtered vocabulary is then ~V = sort_freq(V_all)[1:k] where frequency is calculated over our STARR EHR OMOP dataset.

CLMBR Vocabulary Summary

• 21 Source Ontologies/Knowledge Graphs
• 65,536 tokens (the max value of uint16_t )

Citation

BibTeX:

@article{wornow2023ehrshot,
  title={EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models}, 
  author={Michael Wornow and Rahul Thapa and Ethan Steinberg and Jason Fries and Nigam Shah},
  booktitle={Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
  year={2023}
}

Model Card Authors

Michael Wornow, Ethan Steinberg, Rahul Thapa, Jason Fries, Nigam H. Shah

Model Card Contact

Michael Wornow ( [email protected] )