Sponsored by iMean AI Trip Planner
Toolify Toolify logo
AI Models
Clinical-Longformer

yikuan8 / Clinical-Longformer

Run on huggingface.co
huggingface.co
Total runs: 454.5K
24-hour runs: -4.9K
7-day runs: -9.2K
30-day runs: -33.1K
Model's Last Updated: January 25 2023
fill-mask

Introduction of Clinical-Longformer

Model Details of Clinical-Longformer

Clinical-Longformer is a clinical knowledge enriched version of Longformer that was further pre-trained using MIMIC-III clinical notes. It allows up to 4,096 tokens as the model input. Clinical-Longformer consistently out-performs ClinicalBERT across 10 baseline dataset for at least 2 percent. Those downstream experiments broadly cover named entity recognition (NER), question answering (QA), natural language inference (NLI) and text classification tasks. For more details, please refer to our paper . We also provide a sister model at Clinical-BigBIrd

Pre-training

We initialized Clinical-Longformer from the pre-trained weights of the base version of Longformer. The pre-training process was distributed in parallel to 6 32GB Tesla V100 GPUs. FP16 precision was enabled to accelerate training. We pre-trained Clinical-Longformer for 200,000 steps with batch size of 6×3. The learning rates were 3e-5 for both models. The entire pre-training process took more than 2 weeks.

Usage

Load the model directly from Transformers: 

from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("yikuan8/Clinical-Longformer")
model = AutoModelForMaskedLM.from_pretrained("yikuan8/Clinical-Longformer")
Citing

If you find our model helps, please consider citing this :) 

@article{li2023comparative,
  title={A comparative study of pretrained language models for long clinical text},
  author={Li, Yikuan and Wehbe, Ramsey M and Ahmad, Faraz S and Wang, Hanyin and Luo, Yuan},
  journal={Journal of the American Medical Informatics Association},
  volume={30},
  number={2},
  pages={340--347},
  year={2023},
  publisher={Oxford University Press}
}
Questions

Please email [email protected]

Runs of yikuan8 Clinical-Longformer on huggingface.co

454.5K
Total runs
-4.9K
24-hour runs
-6.5K
3-day runs
-9.2K
7-day runs
-33.1K
30-day runs

More Information About Clinical-Longformer huggingface.co Model

Clinical-Longformer huggingface.co

Clinical-Longformer huggingface.co is an AI model on huggingface.co that provides Clinical-Longformer's model effect (), which can be used instantly with this yikuan8 Clinical-Longformer model. huggingface.co supports a free trial of the Clinical-Longformer model, and also provides paid use of the Clinical-Longformer. Support call Clinical-Longformer model through api, including Node.js, Python, http.

Clinical-Longformer huggingface.co Url

https://huggingface.co/yikuan8/Clinical-Longformer

yikuan8 Clinical-Longformer online free

Clinical-Longformer huggingface.co is an online trial and call api platform, which integrates Clinical-Longformer's modeling effects, including api services, and provides a free online trial of Clinical-Longformer, you can try Clinical-Longformer online for free by clicking the link below.

yikuan8 Clinical-Longformer online free url in huggingface.co:

https://huggingface.co/yikuan8/Clinical-Longformer

Clinical-Longformer install

Clinical-Longformer is an open source model from GitHub that offers a free installation service, and any user can find Clinical-Longformer on GitHub to install. At the same time, huggingface.co provides the effect of Clinical-Longformer install, users can directly use Clinical-Longformer installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

Clinical-Longformer install url in huggingface.co:

https://huggingface.co/yikuan8/Clinical-Longformer

Url of Clinical-Longformer

Clinical-Longformer huggingface.co Url

Provider of Clinical-Longformer huggingface.co

yikuan8
ORGANIZATIONS
huggingface.co

Other API from yikuan8