yosefw / SPLADE-BERT-Mini-BS256

SPLADE-BERT-Mini

This is a SPLADE Sparse Encoder model finetuned from prajjwal1/bert-mini using the sentence-transformers library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

Model Description

• Model Type: SPLADE Sparse Encoder
• Base model: prajjwal1/bert-mini
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 30522 dimensions
• Similarity Function: Dot Product
• Language: en
• License: mit

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertForMaskedLM'})
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("yosefw/SPLADE-BERT-Mini-BS256")
# Run inference
queries = [
    "what paint brands are owned by ppg",
]
documents = [
    'Available throughout the U.S. and Canada at The Home Depot®, Walmart, company-owned stores and other fine retailers, Glidden Paint is a brand of PPG, the leading globalpaint and coatings company.',
    'In 2007, the company was involved in a lawsuit on failing to disclose a purchase reduction of its two major auto glass customers. On 2 January 2008, PPG acquired the SigmaKalon Group of companies for $3.2 billion from private investment firm Bain Capital, strongly increasing its paint and specialty coatings business.',
    'Extraocular Muscles: There are six muscles that are present in the orbit (eye socket) that attach to the eye to move it. These muscles work to move the eye up, down, side to side, and rotate the eye. The superior rectus is an extraocular muscle that attaches to the top of the eye. It moves the eye upward.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 30522] [3, 30522]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[18.9594, 14.0199,  0.0263]])

Evaluation

Metrics

Sparse Information Retrieval

• Evaluated with SparseInformationRetrievalEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 528,247 training samples
• Columns: query , positive , and negative
• Approximate statistics based on the first 1000 samples:

  	query	positive	negative
  type	string	string	string
  details	min: 4 tokens mean: 9.04 tokens max: 28 tokens	min: 20 tokens mean: 79.47 tokens max: 219 tokens	min: 22 tokens mean: 77.35 tokens max: 253 tokens

• Samples:

  query	positive	negative
  how much does a average american spend on for entertainment	How much do Americans spend on entertainment costs? These expenses, like all others, are important to keep track of so you don’t end up not being able to pay off the more important debts. According to the US Department of Labor, the average American family spends a little over $2,500 a year on entertainment or about 5.5% of their earned income. This breaks down to more than $200 a month. How much is your family spending on entertainment costs? Alternatives to high entertainment costs: There is a lot of entertainment outside of the home that is not as pricey and may even be free. Families should take advantage of free events that take place locally. Even driving out of town for a free event or festival can be affordable.	Consumer spending on entertainment by household income in 2013. May 21, 2015. Entertainment has long been a household budget staple. In 2013, the average household spent $2,482 on entertainment, or 4.9 percent of total household spending. Income affects most household spending patterns. In other words, the higher the household income, the greater the dollar amount spent on goods and services in general.
  what is dermatofibroma	Dermatofibroma is a non cancerous skin growth that typically occurs on the legs. They are small in size and generally grow up to less than half an inch or 1 cm in diameter. Dermatofibroma is mainly made up of fibrous tissue. The condition is also known as sclerosing hemangioma. Sponsored link. The skin growth is a type of urticaria, which is a condition characterized by the development of skin welts along the areas where the affected person has been scratched or stroked.	Best Answer: Definition: Dermatofibroma is defined as a benign skin growths, found especially on the legs. They can grow up to about 1 cm (less than a half inch) in diameter. The skin growth is mostly consist of fibrous tissue. Also called sclerosing hemangioma.
  what is a interneurons	An inter neuron is a multi polar neuron, or a neuron with one or more Dendrite. Dendrites are branched projections from the main body, or soma of the nerve cell. Dendrites usually receive information via electrochemical signals from the axon of another neuron, but they can also send out certain types of signals.	What is an Interneuron? An Inter neuron - also known as relay neuron, association neuron, connector neuron, and local circuit neuron - is a multi polar neuron (those that have more than one dendrite) which connects afferent neurons and efferent neurons in neural pathways.

• Loss: SpladeLoss with these parameters:

  {
      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score')",
      "document_regularizer_weight": 0.003,
      "query_regularizer_weight": 0.005
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy : epoch
• per_device_train_batch_size : 32
• per_device_eval_batch_size : 32
• gradient_accumulation_steps : 8
• learning_rate : 8e-05
• num_train_epochs : 10
• lr_scheduler_type : cosine
• warmup_ratio : 0.025
• fp16 : True
• load_best_model_at_end : True
• optim : adamw_torch_fused
• push_to_hub : True
• batch_sampler : no_duplicates

All Hyperparameters

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 5.0.0
• Transformers: 4.53.2
• PyTorch: 2.6.0+cu124
• Accelerate: 1.8.1
• Datasets: 4.0.0
• Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

SparseMultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

FlopsLoss

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
}