Sponsored by Vmake Labs Video Enhancer
Toolify Toolify logo
AI Models
Clinical-BigBird

yikuan8 / Clinical-BigBird

Run on huggingface.co
huggingface.co
Total runs: 787
24-hour runs: 15
7-day runs: 169
30-day runs: -79
Model's Last Updated: April 11 2022
fill-mask

Introduction of Clinical-BigBird

Model Details of Clinical-BigBird

Clinical-BigBird is a clinical knowledge enriched version of BigBird that was further pre-trained using MIMIC-III clinical notes. It allows up to 4,096 tokens as the model input. Clinical-BigBird consistently out-performs ClinicalBERT across 10 baseline dataset. Those downstream experiments broadly cover named entity recognition (NER), question answering (QA), natural language inference (NLI) and text classification tasks. For more details, please refer to our paper . We also provide a sister model at Clinical-Longformer

Pre-training

We initialized Clinical-BigBird from the pre-trained weights of the base version of BigBird. The pre-training process was distributed in parallel to 6 32GB Tesla V100 GPUs. FP16 precision was enabled to accelerate training. We pre-trained Clinical-BigBird for 300,000 steps with batch size of 6×2. The learning rates were 3e-5. The entire pre-training process took more than 2 weeks.

Usage

Load the model directly from Transformers: 

from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("yikuan8/Clinical-BigBird")
model = AutoModelForMaskedLM.from_pretrained("yikuan8/Clinical-BigBird")
Citing

If you find our model helps, please consider citing this :) 

@article{li2022clinical,
  title={Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences},
  author={Li, Yikuan and Wehbe, Ramsey M and Ahmad, Faraz S and Wang, Hanyin and Luo, Yuan},
  journal={arXiv preprint arXiv:2201.11838},
  year={2022}
}
Questions

Please email [email protected]

Runs of yikuan8 Clinical-BigBird on huggingface.co

787
Total runs
15
24-hour runs
99
3-day runs
169
7-day runs
-79
30-day runs

More Information About Clinical-BigBird huggingface.co Model

Clinical-BigBird huggingface.co

Clinical-BigBird huggingface.co is an AI model on huggingface.co that provides Clinical-BigBird's model effect (), which can be used instantly with this yikuan8 Clinical-BigBird model. huggingface.co supports a free trial of the Clinical-BigBird model, and also provides paid use of the Clinical-BigBird. Support call Clinical-BigBird model through api, including Node.js, Python, http.

Clinical-BigBird huggingface.co Url

https://huggingface.co/yikuan8/Clinical-BigBird

yikuan8 Clinical-BigBird online free

Clinical-BigBird huggingface.co is an online trial and call api platform, which integrates Clinical-BigBird's modeling effects, including api services, and provides a free online trial of Clinical-BigBird, you can try Clinical-BigBird online for free by clicking the link below.

yikuan8 Clinical-BigBird online free url in huggingface.co:

https://huggingface.co/yikuan8/Clinical-BigBird

Clinical-BigBird install

Clinical-BigBird is an open source model from GitHub that offers a free installation service, and any user can find Clinical-BigBird on GitHub to install. At the same time, huggingface.co provides the effect of Clinical-BigBird install, users can directly use Clinical-BigBird installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

Clinical-BigBird install url in huggingface.co:

https://huggingface.co/yikuan8/Clinical-BigBird

Url of Clinical-BigBird

Clinical-BigBird huggingface.co Url

Provider of Clinical-BigBird huggingface.co

yikuan8
ORGANIZATIONS
huggingface.co

Other API from yikuan8