swardiantara / bert-tiny-snli-k5-fixed-euclidean

SentenceTransformer based on google/bert_uncased_L-2_H-128_A-2

This is a sentence-transformers model finetuned from google/bert_uncased_L-2_H-128_A-2 . It maps sentences & paragraphs to a 128-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google/bert_uncased_L-2_H-128_A-2
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 128, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("swardiantara/bert-tiny-snli-k5-fixed-euclidean")
# Run inference
sentences = [
    'Children smiling and waving at camera [SEP] The kids are frowning',
    'A room full of girls raising their hands. [SEP] The boys are jumping on the trampoline.',
    'A woman is walking across the street eating a banana, while a man is following with his briefcase. [SEP] the woman is a seductress',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 128]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9991, 0.9962],
#         [0.9991, 1.0000, 0.9956],
#         [0.9962, 0.9956, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 6,043,097 training samples
• Columns: text_a , text_b , and label
• Approximate statistics based on the first 100 samples:

  	text_a	text_b	label
  type	string	string	list
  modality	text	text
  details	min: 13 tokens mean: 22.88 tokens max: 47 tokens	min: 20 tokens mean: 27.54 tokens max: 43 tokens	size: 2 elements

• Samples:

  text_a	text_b	label
  A person on a horse jumps over a broken down airplane. [SEP] A person is training his horse for a competition.	A woman is walking across the street eating a banana, while a man is following with his briefcase. [SEP] the woman is a seductress	[1.0, 0.0]
  A person on a horse jumps over a broken down airplane. [SEP] A person is training his horse for a competition.	A man on stilts is playing a tuba for money on the boardwalk. [SEP] A male street performer with a tuba is playing outside.	[0.0, 0.5]
  A person on a horse jumps over a broken down airplane. [SEP] A person is training his horse for a competition.	A boy is standing next to a car in front of a clothesline. [SEP] The boy is outside.	[0.0, 0.5]

• Loss: main .OrdinalProxyContrastiveLoss

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size : 1024
• learning_rate : 1e-05
• load_best_model_at_end : True

All Hyperparameters

Training Logs

• The bold row denotes the saved checkpoint.

Training Time

• Training : 36.2 minutes
• Evaluation : 2.6 seconds
• Total : 36.3 minutes

Framework Versions

• Python: 3.12.4
• Sentence Transformers: 5.5.1
• Transformers: 5.11.0
• PyTorch: 2.5.1+cu121
• Accelerate: 1.13.0
• Datasets: 2.21.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}