CrabInHoney / urlbert-tiny-v6

CrabInHoney/urlbert-tiny-v6

CrabInHoney/urlbert-tiny-v6 is a lightweight URL encoder based on the ModernBERT architecture, designed for malicious URL detection, phishing classification, and URL feature extraction.

With only 2.03M parameters and a hidden dimension of 128, the model executes very fast simply due to its small footprint and compact embedding size, while remaining competitive on detection tasks.

The base model was trained via knowledge distillation of hidden representations from an ensemble of multiple teacher models, which were intentionally trained specifically for this purpose. Please note that this model is designed to process and analyze URL strings exclusively.

The repository contains the base encoder alongside 10 pre-trained classification heads trained on various public cybersecurity datasets.

Key Specifications

Benchmark Comparison

The models were evaluated on the JPxxx/url-benchmark-dataset

The benchmark contains 3M URLs with a strict 9:1 benign-to-malicious ratio. Evaluation was conducted using site-aware 5-fold cross-validation, ensuring that no second-level domain or IP address appears in more than one fold to eliminate data leakage.

Feature Extraction (Embeddings)

from transformers import AutoModel, AutoTokenizer
import torch

REPO = "CrabInHoney/urlbert-tiny-v6"

tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModel.from_pretrained(REPO)

text = "http://example.com/login-verify-account"
inputs = tok(text, return_tensors="pt")

with torch.no_grad():
    out = model(**inputs)

print(out.last_hidden_state.shape)

Output:

torch.Size([1, 15, 128])

Important Note on Pooling: For optimal downstream performance, it is highly recommended to use a combination of CLS + MEAN pooling (concatenating the [CLS] token representation with the mean representation of all tokens). This strategy yields the most robust and useful signal for feature extraction.

Pretrained Classification Heads

Trained classification heads are stored in the heads/ directory:

Multi-Head Classification Example

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

REPO = "CrabInHoney/urlbert-tiny-v6"

HEADS = [
    "cybersectony_phishing_email_v2",
    "ealvaradob_phishing_dataset",
    "kmack_phishing_urls",
    "malicious_urls_4class",
    "phishbd_2026",
    "phishdestroy_destroylist",
    "phiusiil_phishing_url",
    "snats_url_classifications_clean",
    "url_65lakh",
    "weborganizer_topic_annotations",
]

tok = AutoTokenizer.from_pretrained(REPO)
text = "http://paypal-secure-login.verify-account.com"

for head in HEADS:
    model = AutoModelForSequenceClassification.from_pretrained(
        REPO, subfolder=f"heads/{head}", trust_remote_code=True
    )
    inputs = tok(text, return_tensors="pt", truncation=True, max_length=model.config.max_length)

    with torch.no_grad():
        logits = model(**inputs).logits

    probs = torch.softmax(logits, dim=-1)[0]
    pred = probs.argmax().item()
    print(f"{head:35s} -> {model.config.id2label[pred]:15s} ({probs[pred]:.3f})")

Output:

cybersectony_phishing_email_v2      -> phishing_url    (0.984)
ealvaradob_phishing_dataset         -> phishing        (0.985)
kmack_phishing_urls                 -> phishing        (0.984)
malicious_urls_4class               -> phishing        (0.979)
phishbd_2026                        -> phishing        (0.985)
phishdestroy_destroylist            -> malicious       (0.984)
phiusiil_phishing_url               -> phishing        (0.985)
snats_url_classifications_clean     -> legal           (0.352)
url_65lakh                          -> malicious       (0.985)
weborganizer_topic_annotations      -> Finance & Business (0.485)