swardiantara / bert-tiny-snli-k1-fixed-euclidean

SentenceTransformer based on google/bert_uncased_L-2_H-128_A-2

This is a sentence-transformers model finetuned from google/bert_uncased_L-2_H-128_A-2 . It maps sentences & paragraphs to a 128-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google/bert_uncased_L-2_H-128_A-2
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 128, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("swardiantara/bert-tiny-snli-k1-fixed-euclidean")
# Run inference
sentences = [
    'Bicyclists waiting at an intersection. [SEP] The bicyclists are in a race.',
    'A young shirtless boy in kakhi pants is kneeling in a marsh while someone splashes nearby. [SEP] Two people are riding on a ferris wheel at the fair.',
    'A woman in a top hat is trying to get into a maroon car at night. [SEP] The woman and the car are outdoors.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 128]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9997, 0.9998],
#         [0.9997, 1.0000, 0.9997],
#         [0.9998, 0.9997, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,648,101 training samples
• Columns: text_a , text_b , and label
• Approximate statistics based on the first 100 samples:

  	text_a	text_b	label
  type	string	string	list
  modality	text	text
  details	min: 13 tokens mean: 27.21 tokens max: 47 tokens	min: 27 tokens mean: 30.28 tokens max: 36 tokens	size: 2 elements

• Samples:

  text_a	text_b	label
  A person on a horse jumps over a broken down airplane. [SEP] A person is training his horse for a competition.	A woman standing in front of a white car that is piled with things on top. [SEP] The woman is preparing to move.	[1.0, 0.0]
  A person on a horse jumps over a broken down airplane. [SEP] A person is training his horse for a competition.	A woman in a top hat is trying to get into a maroon car at night. [SEP] The woman and the car are outdoors.	[0.0, 0.5]
  A person on a horse jumps over a broken down airplane. [SEP] A person is training his horse for a competition.	A young shirtless boy in kakhi pants is kneeling in a marsh while someone splashes nearby. [SEP] Two people are riding on a ferris wheel at the fair.	[0.0, 0.5]

• Loss: main .OrdinalProxyContrastiveLoss

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size : 1024
• learning_rate : 1e-05
• load_best_model_at_end : True

All Hyperparameters

Training Logs

• The bold row denotes the saved checkpoint.

Training Time

• Training : 9.4 minutes
• Evaluation : 2.5 seconds
• Total : 9.4 minutes

Framework Versions

• Python: 3.12.4
• Sentence Transformers: 5.5.1
• Transformers: 5.11.0
• PyTorch: 2.5.1+cu121
• Accelerate: 1.13.0
• Datasets: 2.21.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}