swardiantara / bert-tiny-snli-k3-adaptive-euclidean

SentenceTransformer based on google/bert_uncased_L-2_H-128_A-2

This is a sentence-transformers model finetuned from google/bert_uncased_L-2_H-128_A-2 . It maps sentences & paragraphs to a 128-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google/bert_uncased_L-2_H-128_A-2
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 128, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("swardiantara/bert-tiny-snli-k3-adaptive-euclidean")
# Run inference
sentences = [
    'A couple play in the tide with their young son. [SEP] The family is on vacation.',
    'A woman with a panda hat and headphones is in front of a man outside in the snow. [SEP] The man is on the beach.',
    'Male standing in the roadway, he is wearing a light colored shirt and holding a backpack. [SEP] A male is watching his dogs play.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 128]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9980, 0.9919],
#         [0.9980, 1.0000, 0.9903],
#         [0.9919, 0.9903, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 3,845,587 training samples
• Columns: text_a , text_b , and label
• Approximate statistics based on the first 100 samples:

  	text_a	text_b	label
  type	string	string	list
  modality	text	text
  details	min: 13 tokens mean: 24.67 tokens max: 47 tokens	min: 22 tokens mean: 28.88 tokens max: 43 tokens	size: 2 elements

• Samples:

  text_a	text_b	label
  A person on a horse jumps over a broken down airplane. [SEP] A person is training his horse for a competition.	A man in goggles and a hat is sitting in the street holding something and wearing white gloves. [SEP] The man is a performance artist.	[1.0, 0.0]
  A person on a horse jumps over a broken down airplane. [SEP] A person is training his horse for a competition.	Two boys and a woman standing in front of a Pronto Pups Hamburger stand. [SEP] People are outside.	[0.0, 0.5]
  A person on a horse jumps over a broken down airplane. [SEP] A person is training his horse for a competition.	A little girl in a red shirt is suspended in midair on a contraption, as a man wearing white shorts looks up at her. [SEP] A little girl and a man are looking at each other.	[0.0, 0.5]

• Loss: main .OrdinalProxyContrastiveLoss

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size : 1024
• learning_rate : 1e-05
• load_best_model_at_end : True

All Hyperparameters

Training Logs

• The bold row denotes the saved checkpoint.

Training Time

• Training : 22.9 minutes
• Evaluation : 2.5 seconds
• Total : 22.9 minutes

Framework Versions

• Python: 3.12.4
• Sentence Transformers: 5.5.1
• Transformers: 5.11.0
• PyTorch: 2.5.1+cu121
• Accelerate: 1.13.0
• Datasets: 2.21.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}