dleemiller / ModernCE-base-sts

ModernBERT Cross-Encoder: Semantic Similarity (STS)

Cross encoders are high performing encoder models that compare two texts and output a 0-1 score. I've found the cross-encoders/roberta-large-stsb model to be very useful in creating evaluators for LLM outputs. They're simple to use, fast and very accurate.

Like many people, I was excited about the architecture and training uplift from the ModernBERT architecture ( answerdotai/ModernBERT-base ). So I've applied it to the stsb cross encoder, which is a very handy model. Additionally, I've added pretraining from my much larger semi-synthetic dataset dleemiller/wiki-sim that targets this kind of objective. The inference performance efficiency, expanded context and simplicity make this a really nice platform as an evaluator model.

---

Features

• High performing: Achieves Pearson: 0.9162 and Spearman: 0.9122 on the STS-Benchmark test set.
• Efficient architecture: Based on the ModernBERT-base design (149M parameters), offering faster inference speeds.
• Extended context length: Processes sequences up to 8192 tokens, great for LLM output evals.
• Diversified training: Pretrained on dleemiller/wiki-sim and fine-tuned on sentence-transformers/stsb .

---

Performance

---

Usage

To use ModernCE for semantic similarity tasks, you can load the model with the Hugging Face sentence-transformers library:

from sentence_transformers import CrossEncoder

# Load ModernCE model
model = CrossEncoder("dleemiller/ModernCE-base-sts")

# Predict similarity scores for sentence pairs
sentence_pairs = [
    ("It's a wonderful day outside.", "It's so sunny today!"),
    ("It's a wonderful day outside.", "He drove to work earlier."),
]
scores = model.predict(sentence_pairs)

print(scores)  # Outputs: array([0.9184, 0.0123], dtype=float32)

Output

The model returns similarity scores in the range [0, 1] , where higher scores indicate stronger semantic similarity.

---

Training Details

Pretraining

The model was pretrained on the pair-score-sampled subset of the dleemiller/wiki-sim dataset. This dataset provides diverse sentence pairs with semantic similarity scores, helping the model build a robust understanding of relationships between sentences.

• Classifier Dropout: a somewhat large classifier dropout of 0.3, to reduce overreliance on teacher scores.
• Objective: STS-B scores from cross-encoder/stsb-roberta-large .

Fine-Tuning

Fine-tuning was performed on the sentence-transformers/stsb dataset.

Validation Results

The model achieved the following test set performance after fine-tuning:

• Pearson Correlation: 0.9162
• Spearman Correlation: 0.9122

Logs for training and evaluation are included in the training logs .

---

Applications

1. Semantic Search: Retrieve relevant documents or text passages based on query similarity.
2. Retrieval-Augmented Generation (RAG): Enhance generative models by providing contextually relevant information.
3. Content Moderation: Automatically classify or rank content based on similarity to predefined guidelines.
4. Code Search: Leverage the model's ability to understand code and natural language for large-scale programming tasks.

---

Model Card

• Architecture: ModernBERT-base
• Tokenizer: Custom tokenizer trained with modern techniques for long-context handling.
• Pretraining Data: dleemiller/wiki-sim (pair-score-sampled)
• Fine-Tuning Data: sentence-transformers/stsb

---

Thank You

Thanks to the AnswerAI team for providing the ModernBERT models, and the Sentence Transformers team for their leadership in transformer encoder models.

---

Citation

If you use this model in your research, please cite:

@misc{moderncestsb2025,
  author = {Miller, D. Lee},
  title = {ModernCE STS: An STS cross encoder model},
  year = {2025},
  publisher = {Hugging Face Hub},
  url = {https://huggingface.co/dleemiller/ModernCE-base-sts},
}

---

License

This model is licensed under the MIT License .