YeonwooSung / nomic-embed-text-v2-moe-msmarco-bpr

SentenceTransformer based on nomic-ai/nomic-embed-text-v2-moe

This is a sentence-transformers model finetuned from nomic-ai/nomic-embed-text-v2-moe . It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/nomic-embed-text-v2-moe
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: NomicBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("BlackBeenie/nomic-embed-text-v2-moe-msmarco-bpr")
# Run inference
sentences = [
    'what services are offered by adult day care',
    'Consumer Guide to Long Term Care. Adult Day Care. Adult day care is a planned program offered in a group setting that provides services that improve or maintain health or functioning, and social activities for seniors and persons with disabilities.',
    'The Met Life Market survey of 2008 on adult day services states the average cost for adult day care services is $64 per day. There has been an increase of 5% in these services in the past year.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 498,970 training samples
• Columns: sentence_0 , sentence_1 , and sentence_2
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	sentence_2
  type	string	string	string
  details	min: 4 tokens mean: 9.75 tokens max: 24 tokens	min: 24 tokens mean: 89.23 tokens max: 241 tokens	min: 20 tokens mean: 86.66 tokens max: 280 tokens

• Samples:

  sentence_0	sentence_1	sentence_2
  what the history of bluetooth	When asked about the name Bluetooth, I explained that Bluetooth was borrowed from the 10th century, second King of Denmark, King Harald Bluetooth; who was famous for uniting Scandinavia just as we intended to unite the PC and cellular industries with a short-range wireless link.	Technology: 1 How secure is a Bluetooth network? 2 What is Frequency-Hopping Spread Spectrum (FHSS)? 3 Will other RF (Radio Frequency) devices interfere with Bluetooth Devices? 4 Will Bluetooth and Wireless LAN (WLAN) interfere with each other? 5 What is the data throughput speed of a Bluetooth connection? 6 What is the range of Bluetooth 7 ... What kind of ...
  how thin can a concrete slab be	Another issue that must be addressed is the added weight of the thin-slab. Poured gypsum thin-slabs typically add 13 to 15 pounds per square foot to the dead loading of a floor structure. Standard weight concrete thin slabs add about 18 pounds per square foot (at 1.5 thickness).	Find the Area in square feet: We will use a concrete slab pour for our example. Let’s say that we need to figure out the yardage for a slab that will be 15 feet long by 10 feet wide and 4 inches thick. First we find the area by multiplying the length times the width. 1 15 feet X 10 feet = 150 square feet.
  how long to cook eggs to hard boil	This method works best if the eggs are in a single layer, but you can double them up as well, you'll just need to add more time to the steaming time. 3 Set your timer for 6 minutes for soft boiled, 10 minutes for hard boiled with a still translucent and bright yolk, or 12-15 minutes for cooked-through hard boiled.	Hard-Steamed Eggs. Fill a pot that can comfortably hold your steamer with the lid on with 1 to 2 inches of water. Bring to a rolling boil, 212 degrees Fahrenheit. Place your eggs in a metal steamer, and lower the basket into the pot. The eggs should sit above the boiling water. Cover and cook for 12 minutes. Hard-steamed eggs, like hard-boiled eggs, are eggs that are cooked until the egg yolk is fully set and has turned to a chalky texture.

• Loss: beir.losses.bpr_loss.BPRLoss

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy : steps
• per_device_train_batch_size : 32
• per_device_eval_batch_size : 32
• num_train_epochs : 5
• fp16 : True
• multi_dataset_batch_sampler : round_robin

All Hyperparameters

Training Logs

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.4.1
• Transformers: 4.49.0
• PyTorch: 2.5.1+cu124
• Accelerate: 1.3.0
• Datasets: 3.3.2
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}