cisco-ai / SecureBERT2.0-base

Model Card for cisco-ai/SecureBERT2.0-base

SecureBERT 2.0 Base is a domain-specific transformer model optimized for cybersecurity tasks. It extends the ModernBERT architecture with cybersecurity-focused pretraining to produce contextualized embeddings for both technical text and code. SecureBERT 2.0 supports tasks like masked language modeling, semantic search, named entity recognition, vulnerability detection, and code analysis.

---

Model Details

Model Description

SecureBERT 2.0 Base is designed for deep contextual understanding of cybersecurity language and code . It leverages domain-specific pretraining on a large, heterogeneous corpus covering threat reports, blogs, documentation, and codebases, making it effective for reasoning across natural language and programming syntax.

• Developed by: Cisco AI
• Model type: Transformer (ModernBERT architecture)
• Language: English
• License: Apache 2.0
• Finetuned from model: answerdotai/ModernBERT-base

Model Sources

• Repository: https://huggingface.co/cisco-ai/SecureBERT2.0-base
• Paper: arXiv:2510.00240

---

Uses

Direct Use

• Masked language modeling for cybersecurity text and code
• Embedding generation for semantic search and retrieval
• Code and text feature extraction for downstream classification or clustering
• Named entity recognition (NER) on security-related entities
• Vulnerability detection in source code

Downstream Use

Fine-tuning for:

• Threat intelligence extraction
• Security question answering
• Incident analysis and summarization
• Automated code review and vulnerability prediction

Out-of-Scope Use

• Non-English or non-technical text
• General-purpose conversational AI
• Decision-making in real-time security systems without human oversight

---

Bias, Risks, and Limitations

The model reflects biases in the cybersecurity sources it was trained on, which may include:

• Overrepresentation of certain threat actors, technologies, or organizations
• Inconsistent code or documentation quality
• Limited exposure to non-public or proprietary data formats

Recommendations

Users should evaluate outputs in their specific context and avoid automated high-stakes decisions without expert validation.

---

How to Get Started with the Model

from transformers import AutoModelForMaskedLM, AutoTokenizer

model_name = "cisco-ai/SecureBERT2.0-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)

text = "The malware exploits a vulnerability in the [MASK] system."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

predicted_token_id = outputs.logits.argmax(-1)
predicted_word = tokenizer.decode(predicted_token_id[0])
print(predicted_word)

Training Details

Training Procedure

Preprocessing

Hybrid tokenization for text and code (natural language + structured syntax).

Training Hyperparameters

• Objective: Masked Language Modeling (MLM)
• Masking probability: 0.10
• Optimizer: AdamW
• Learning rate: 5e-5
• Weight decay: 0.01
• Epochs: 20
• Batch size: 16 per GPU × 8 GPUs
• Curriculum: Microannealing (gradual dataset diversification)

---

Evaluation

Testing Data, Factors & Metrics

Testing Data

Internal held-out subset of cybersecurity and code corpora.

Factors

Evaluated across token categories:

• Objects (nouns)
• Actions (verbs)
• Code tokens

Metrics

Top-n accuracy on masked token prediction.

Results

This figure presents a comparative study of SecureBERT 2.0, SecureBERT, and ModernBERT on the masked language modeling (MLM) task. This shows SecureBERT 2.0 outperforms both the original SecureBERT and generic ModernBERT, particularly in code understanding and domain-specific terms.

Summary

SecureBERT 2.0 outperforms both the original SecureBERT and ModernBERT on cybersecurity-specific and code-related tasks.

---

Environmental Impact

• Hardware Type: 8× GPU cluster
• Hours used: [Information Not Available]
• Cloud Provider: [Information Not Available]
• Compute Region: [Information Not Available]
• Carbon Emitted: [Estimate Not Available]

Carbon footprint can be estimated using Lacoste et al. (2019) .

---

Technical Specifications

Model Architecture and Objective

• Architecture: ModernBERT
• Max sequence length: 1024 tokens
• Parameters: 150 M
• Objective: Masked Language Modeling (MLM)
• Tensor type: F32

Compute Infrastructure

• Framework: Transformers (PyTorch)
• Mixed Precision: fp32
• Hardware: 8 GPUs
• Checkpoint Format: Safetensors

Citation

BibTeX:

@article{aghaei2025securebert,
  title={SecureBERT 2.0: Advanced Language Model for Cybersecurity Intelligence},
  author={Aghaei, Ehsan and Jain, Sarthak and Arun, Prashanth and Sambamoorthy, Arjun},
  journal={arXiv preprint arXiv:2510.00240},
  year={2025}
}

APA:

Cisco AI (2025). SecureBERT 2.0: A Domain-Specific Transformer for Cybersecurity and Code Understanding. arXiv:2510.00240.

---

Model Card Authors

Cisco AI

Model Card Contact

For inquiries, please contact [email protected]