cisco-ai / SecureBERT2.0-biencoder

Model Card for cisco-ai/SecureBERT2.0-biencoder

The SecureBERT 2.0 Bi-Encoder is a cybersecurity-domain sentence-similarity and document-embedding model fine-tuned from SecureBERT 2.0 .
It independently encodes queries and documents into a shared vector space for semantic search , information retrieval , and cybersecurity knowledge retrieval .

---

Model Details

Model Description

• Developed by: Cisco AI
• Model type: Bi-Encoder (Sentence Transformer)
• Architecture: ModernBERT backbone with dual encoders
• Max sequence length: 1024 tokens
• Output dimension: 768
• Language: English
• License: Apache-2.0
• Finetuned from: cisco-ai/SecureBERT2.0-base

---

Uses

Direct Use

• Semantic search and document similarity in cybersecurity corpora
• Information retrieval and ranking for threat intelligence reports, advisories, and vulnerability notes
• Document embedding for retrieval-augmented generation (RAG) and clustering

Downstream Use

• Threat intelligence knowledge graph construction
• Cybersecurity QA and reasoning systems
• Security operations center (SOC) data mining

Out-of-Scope Use

• Non-technical or general-domain text similarity
• Generative or conversational tasks

---

Model Architecture

The Bi-Encoder encodes queries and documents independently into a joint vector space.
This architecture enables scalable approximate nearest-neighbor search for candidate retrieval and semantic ranking.

---

Datasets

Fine-Tuning Datasets

Dataset Descriptions

• Cybersecurity QA corpus: 43 k question–answer pairs, reports, and technical documents covering network security, malware analysis, cryptography, and cloud security.
• Security governance QA corpus: 60 k expert-curated governance and compliance QA pairs emphasizing clear, validated responses.
• Cybersecurity instruction–response corpus: 25 k instructional pairs enabling reasoning and instruction-following.
• Cybersecurity rules corpus: 5 k structured policy and guideline records used for evaluation.

---

How to Get Started with the Model

Using Sentence Transformers

pip install -U sentence-transformers

Run Model to Encode

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("cisco-ai/SecureBERT2.0-biencoder")

sentences = [
    "How would you use Amcache analysis to detect fileless malware?",
    "Amcache analysis provides forensic artifacts for detecting fileless malware ...",
    "To capture and display network traffic"
]

embeddings = model.encode(sentences)
print(embeddings.shape)

Compute Similarity

from sentence_transformers import util
similarity = util.cos_sim(embeddings, embeddings)
print(similarity)

---

Framework Versions

• python: 3.10.10
• sentence_transformers: 5.0.0
• transformers: 4.52.4
• PyTorch: 2.7.0+cu128
• accelerate: 1.9.0
• datasets: 3.6.0

---

Training Details

Training Dataset

The model was fine-tuned on cybersecurity-specific paired-sentence data for document embedding and similarity learning.

• Dataset Size: 35,705 samples
• Columns: sentence_0 , sentence_1 , label

Example Schema

Example Samples

---

Training Objective and Loss

The model was optimized to maximize semantic similarity between relevant cybersecurity text pairs using contrastive learning.

• Loss Function: MultipleNegativesRankingLoss

Loss Parameters

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

Reference

@article{aghaei2025securebert,
  title={SecureBERT 2.0: Advanced Language Model for Cybersecurity Intelligence},
  author={Aghaei, Ehsan and Jain, Sarthak and Arun, Prashanth and Sambamoorthy, Arjun},
  journal={arXiv preprint arXiv:2510.00240},
  year={2025}
}

---

Model Card Authors

Cisco AI

Model Card Contact

For inquiries, please contact [email protected]