bobox / DeBERTa-small-ST-v1-test-UnifiedDatasets

SentenceTransformer based on bobox/DeBERTa-small-ST-v1-test-step3

This is a sentence-transformers model finetuned from bobox/DeBERTa-small-ST-v1-test-step3 on the bobox/enhanced_nli-50_k dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: bobox/DeBERTa-small-ST-v1-test-step3
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Training Dataset:
  • bobox/enhanced_nli-50_k

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: DebertaV2Model 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("bobox/DeBERTa-small-ST-v1-test-UnifiedDatasets")
# Run inference
sentences = [
    'can an object have constant non zero velocity and changing acceleration?',
    'Acceleration is defined as the time derivative of the velocity; if the velocity is unchanging the acceleration is zero. Velocity is a vector, speed is a scalar magnitude of the vector. If the velocity vector changes direction you can have constant speed (not velocity) with a non-zero acceleration.',
    'Acne treatment is individual and customized to the type of acne you have. On average, mild acne responds in 1-2 months, moderate acne responds in 2-4 months and severe acne can take 4-6 months to clear, granted that the most effective measures can be used.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Semantic Similarity

• Dataset: sts-test
• Evaluated with EmbeddingSimilarityEvaluator

Training Details

Training Dataset

bobox/enhanced_nli-50_k

• Dataset: bobox/enhanced_nli-50_k
• Size: 120,849 training samples
• Columns: sentence1 and sentence2
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2
  type	string	string
  details	min: 4 tokens mean: 32.01 tokens max: 336 tokens	min: 2 tokens mean: 60.45 tokens max: 512 tokens

• Samples:

  sentence1	sentence2
  A lady working in a kitchen with several different types of dishes.	A woman is cooking and cleaning in her kitchen.
  can you renew your licence online sa?	You can renew your licence online for as long as your photo is valid. Renew your driver's licence online with a mySA GOV account. With a mySA GOV account, you can access a legally compliant digital licence through the mySA GOV app.
  how can coconut oil lower cholesterol	It has been shown that lauric acid increases the good HDL cholesterol in the blood to help improve cholesterol ratio levels. Coconut oil lowers cholesterol by promoting its conversion to pregnenolone, a molecule that is a precursor to many of the hormones our bodies need. Coconut can help restore normal thyroid function. When the thyroid does not function optimally, it can contribute to higher levels of bad cholesterol.

• Loss: CachedGISTEmbedLoss with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.025}
  

Evaluation Dataset

bobox/enhanced_nli-50_k

• Dataset: bobox/enhanced_nli-50_k
• Size: 3,052 evaluation samples
• Columns: sentence1 and sentence2
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2
  type	string	string
  details	min: 4 tokens mean: 32.91 tokens max: 342 tokens	min: 2 tokens mean: 60.3 tokens max: 408 tokens

• Samples:

  sentence1	sentence2
  The body was found in the River Avon in Bath, Avon and Somerset Police said. Officers said although formal identification had not yet taken place, Henry Burke's family had been told. Earlier officers said they were looking for Mr Burke, who was last seen leaving a nightclub in George Street late on Thursday. A force spokesman said the death was being treated as unexplained and inquiries were continuing. Mr Burke's girlfriend, Em Comley, earlier said he had been texting her "throughout the night" but then the messages suddenly stopped just after midnight.	A man's body has been found in a river after search and rescue teams were called in to try and find a missing 19-year-old student.
  what happens when the president of united states is impeached?	Parliament votes on the proposal by secret ballot, and if two thirds of all representatives agree, the president is impeached. Once impeached, the president's powers are suspended, and the Constitutional Court decides whether or not the President should be removed from office.
  What can feed at more than one trophic level?	Many consumers feed at more than one trophic level.. Nuts are also consumed by deer, turkey, foxes, wood ducks and squirrels. wood ducks can feed at more than one trophic level

• Loss: CachedGISTEmbedLoss with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.025}
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy : steps
• per_device_train_batch_size : 960
• per_device_eval_batch_size : 128
• learning_rate : 3.5e-05
• weight_decay : 0.0001
• num_train_epochs : 2
• lr_scheduler_type : cosine_with_min_lr
• lr_scheduler_kwargs : {'num_cycles': 0.5, 'min_lr': 5.833333333333333e-06}
• warmup_ratio : 0.25
• save_safetensors : False
• fp16 : True
• push_to_hub : True
• hub_model_id : bobox/DeBERTa-small-ST-v1-test-UnifiedDatasets-checkpoints-tmp
• hub_strategy : all_checkpoints
• batch_sampler : no_duplicates

All Hyperparameters

Training Logs

Framework Versions

• Python: 3.10.14
• Sentence Transformers: 3.0.1
• Transformers: 4.44.0
• PyTorch: 2.4.0
• Accelerate: 0.33.0
• Datasets: 2.21.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}