amentaphd / snowflake_artic_embed_l

SentenceTransformer based on Snowflake/snowflake-arctic-embed-l

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-l . It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Snowflake/snowflake-arctic-embed-l
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'What should the determination of the share of the volume auctioned strive to achieve, if appropriate and feasible?',
    'and that those benefits outweigh the effects on the cost and speed of deployment of energy from renewable sources. The determination of the share of the volume auctioned should, if appropriate and feasible, strive to reach a gradual overall increase with a view to contributing to meeting the overall objectives of this Regulation and to reaching 50 % by the end of 2029.',
    '12.\n\nWhere assets and liabilities included in consolidated financial statements have been measured by undertakings included in the consolidation using bases differing from those used for the purposes of the consolidation, those assets and liabilities shall be re-measured in accordance with the bases used for the consolidation. Departures from this requirement shall be permitted in exceptional cases. Any such departures shall be disclosed in the notes to the consolidated financial statements and reasons given.\n\n13.\n\nDeferred tax balances shall be recognised on consolidation provided that it is probable that a charge to tax will arise within the foreseeable future for one of the undertakings included in the consolidation.\n\n14.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 21,478 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 11 tokens mean: 23.55 tokens max: 69 tokens	min: 3 tokens mean: 123.87 tokens max: 445 tokens

• Samples:

  sentence_0	sentence_1
  Which Commission Decisions are to be replaced by the new implementing act related to plastic carrier bags?	and the format for the submission of those data, as those data are necessary to support the monitoring and full implementation of the substantive requirements related to plastic carrier bags, in particular to ensure disaggregated and mandatory data on different categories of plastic carrier bags. That implementing act should replace Commission Decision 2005/270/EC (44) and Commission Implementing Decision (EU) 2018/896 (45).
  What factors should the Commission consider when assessing the progress of Member States in renovating public bodies' buildings during the next review of Directive 2010/31/EU?	be able to apply less stringent requirements to some buildings, such as buildings with special architectural or historical merit. During the next review of Directive 2010/31/EU, the Commission should assess the progress Member States achieved regarding the renovation of public bodies’ buildings. The Commission should consider submitting a legislative proposal to revise the renovation rate, while taking into account the progress achieved by the Member States, substantial economic or technical developments, or where needed, the Union’s commitments for decarbonisation and zero pollution. The obligation to renovate public bodies’ buildings in this Directive complements that in Directive 2010/31/EU, which requires Member States to ensure that
  Which directive is being amended by Directive 2012/6/EU as mentioned in the context?	8. Member States which at 19 July 2013 have brought into force laws, regulations or administrative provisions in compliance with Directive 2012/6/EU of the European Parliament and of the Council of 14 March 2012 amending Council Directive 78/660/EEC on the annual accounts of certain types of companies as regards micro-entities ( 31 ), may be exempted from the requirements set out in Article 3(9) with regard to the conversion into national currencies of thresholds set out in Article 3(1) when applying the first sentence of Article 53(1). 9.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy : steps
• multi_dataset_batch_sampler : round_robin

All Hyperparameters

Training Logs

Framework Versions

• Python: 3.10.15
• Sentence Transformers: 3.4.1
• Transformers: 4.49.0
• PyTorch: 2.6.0+cu126
• Accelerate: 1.5.2
• Datasets: 3.4.1
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}