llm-semantic-router / mmbert-embed-medical

SentenceTransformer based on llm-semantic-router/mmbert-embed-32k-2d-matryoshka

This is a sentence-transformers model finetuned from llm-semantic-router/mmbert-embed-32k-2d-matryoshka . It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: llm-semantic-router/mmbert-embed-32k-2d-matryoshka
• Maximum Sequence Length: 32768 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 32768, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'What is (are) Turner Syndrome ?',
    'Assisted reproduction techniques can help some women with Turner syndrome get pregnant.   NIH: National Institute of Child Health and Human Development',
    "What are the signs and symptoms of Turner syndrome?There are various signs and symptoms of Turner syndrome, which can range from very mild to more severe.Short stature is the most common feature and usually becomes apparent by age 5.In early childhood, frequent middle ear infections are common and can lead to hearing loss in some cases.Most affected girls do not produce the necessary sex hormones for puberty, so they don't have a pubertal growth spurt, start their periods or develop breasts without hormone treatment.While most affected women are infertile, pregnancy is possible with egg donation and assisted reproductive technology.Intelligence is usually normal, but developmental delay, learning disabilities, and/or behavioral problems are sometimes present.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.7383, 0.4883],
#         [0.7383, 1.0000, 0.5508],
#         [0.4883, 0.5508, 1.0000]], dtype=torch.bfloat16)

Training Details

Training Dataset

Unnamed Dataset

• Size: 21,344 training samples
• Columns: sentence_0 , sentence_1 , and sentence_2
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	sentence_2
  type	string	string	string
  details	min: 6 tokens mean: 13.33 tokens max: 27 tokens	min: 12 tokens mean: 148.31 tokens max: 439 tokens	min: 9 tokens mean: 141.22 tokens max: 523 tokens

• Samples:

  sentence_0	sentence_1	sentence_2
  What is the outlook for Coffin Lowry Syndrome ?	The prognosis for individuals with Coffin-Lowry syndrome varies depending on the severity of symptoms. Early intervention may improve the outlook for patients. Life span is reduced in some individuals with Coffin-Lowry syndrome.	How might Coffin-Siris syndrome be treated? People with Coffin-Siris syndrome may benefit from occupational, physical, and speech therapy. Developmental pediatricians may be helpful in recommending and coordinating therapeutic and educational interventions. Additional specialty care may be needed depending on the symptoms in the individual, such as by gastrointestinal, eye, kidney, heart, and hearing specialists.
  What is (are) Sarcoidosis ?	Sarcoidosis is an inflammatory disease characterized by the development and growth of tiny lumps of cells called granulomas. If these tiny granulomas grow and clump together in an organ, they can affect how the organ works, leading to the symptoms of sarcoidosis. The granulomas can be found in almost any part of the body, but occur more commonly in the lungs, lymph nodes, eyes, skin, and liver. Although no one is sure what causes sarcoidosis, it is thought by most scientists to be a disorder of the immune system. The course of the disease varies from person to person. It often goes away on its own, but in some people symptoms of sarcoidosis may last a lifetime. For those who need treatment, anti-inflammatory medications and immunosuppressants can help.	Sarcoidosis is a disease that leads to inflammation, usually in your lungs, skin, or lymph nodes. It starts as tiny, grain-like lumps, called granulomas. Sarcoidosis can affect any organ in your body. No one is sure what causes sarcoidosis. It affects men and women of all ages and races. It occurs mostly in people ages 20 to 50, African Americans, especially women, and people of Northern European origin. Many people have no symptoms. If you have symptoms, they may include - Cough - Shortness of breath - Weight loss - Night sweats - Fatigue Tests to diagnose sarcoidosis include chest x-rays, lung function tests, and a biopsy. Not everyone who has the disease needs treatment. If you do, prednisone, a type of steroid, is the main treatment. NIH: National Heart, Lung, and Blood Institute
  What are the symptoms of High Blood Pressure ?	Because diagnosis is based on blood pressure readings, this condition can go undetected for years, as symptoms do not usually appear until the body is damaged from chronic high blood pressure. Complications of High Blood Pressure When blood pressure stays high over time, it can damage the body and cause complications.Some common complications and their signs and symptoms include: Aneurysms:When an abnormal bulge forms in the wall of an artery.Aneurysms develop and grow for years without causing signs or symptoms until they rupture, grow large enough to press on nearby body parts, or block blood flow.The signs and symptoms that develop depend on the location of the aneurysm. Chronic Kidney Disease: When blood vessels narrow in the kidneys, possibly causing kidney failure. Cognitive Changes: Research shows that over time, higher blood pressure numbers can lead to cognitive changes.	High blood pressure is a common disease in which blood flows through blood vessels (arteries) at higher than normal pressures. There are two main types of high blood pressure: primary and secondary high blood pressure. Primary, or essential, high blood pressure is the most common type of high blood pressure. This type of high blood pressure tends to develop over years as a person ages. Secondary high blood pressure is caused by another medical condition or use of certain medicines. This type usually resolves after the cause is treated or removed.

• Loss: TripletLoss with these parameters:

  {
      "distance_metric": "TripletDistanceMetric.COSINE",
      "triplet_margin": 0.1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• num_train_epochs : 2
• multi_dataset_batch_sampler : round_robin

All Hyperparameters

Training Logs

Framework Versions

• Python: 3.12.3
• Sentence Transformers: 5.2.2
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.12.0
• Datasets: 4.5.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}