bobox / E5-base-unsupervised-TSDAE-2

SentenceTransformer based on intfloat/e5-base-unsupervised

This is a sentence-transformers model finetuned from intfloat/e5-base-unsupervised . It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: intfloat/e5-base-unsupervised
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("bobox/E5-base-unsupervised-TSDAE-2")
# Run inference
sentences = [
    'ligand ion channels located?',
    'where are ligand gated ion channels located?',
    "Duvets tend to be warm but surprisingly lightweight. The duvet cover makes it easier to change bedding looks and styles. You won't need to wash your duvet very often, just wash the cover regularly. Additionally, duvets tend to be fluffier than comforters, and can simplify bed making if you choose the European style.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Semantic Similarity

• Dataset: sts-test
• Evaluated with EmbeddingSimilarityEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 700,000 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 3 tokens mean: 15.73 tokens max: 55 tokens	min: 8 tokens mean: 36.05 tokens max: 131 tokens

• Samples:

  sentence_0	sentence_1
  Quality such a has components with applicable high objective system measure component improvements	Quality in such a system has three components: high accuracy, compliance with applicable standards, and high customer satisfaction. The objective of the system is to measure each component and achieve improvements.
  include	does qbi include capital gains?
  They have a . parietal is in, as becomes and pigments after four to is believed and in circadian cycles	They have a third eye. The parietal eye is only visible in hatchlings, as it becomes covered in scales and pigments after four to six months. Its function is a subject of ongoing research, but it is believed to be useful in absorbing ultraviolet rays and in setting circadian and seasonal cycles.

• Loss: DenoisingAutoEncoderLoss

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy : steps
• per_device_train_batch_size : 16
• per_device_eval_batch_size : 16
• num_train_epochs : 2
• multi_dataset_batch_sampler : round_robin

All Hyperparameters

Training Logs

Framework Versions

• Python: 3.10.13
• Sentence Transformers: 3.0.1
• Transformers: 4.41.2
• PyTorch: 2.1.2
• Accelerate: 0.31.0
• Datasets: 2.19.2
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

DenoisingAutoEncoderLoss

@inproceedings{wang-2021-TSDAE,
    title = "TSDAE: Using Transformer-based Sequential Denoising Auto-Encoderfor Unsupervised Sentence Embedding Learning",
    author = "Wang, Kexin and Reimers, Nils and Gurevych, Iryna", 
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2021",
    month = nov,
    year = "2021",
    address = "Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    pages = "671--688",
    url = "https://arxiv.org/abs/2104.06979",
}