Akash-Sakala / bert-phishing-classifier_student

DistilBERT Phishing Site Classifier (Student)

A 4-layer DistilBERT trained via knowledge distillation from a fine-tuned BERT teacher ( Akash-Sakala/bert-phishing-classifier_teacher ) for binary phishing site URL classification.

Model Details

Training — Distillation Setup

Loss Function

Combined KL divergence (soft targets) + Cross-Entropy (hard labels):

loss = alpha * KL(student_soft || teacher_soft) * T^2 + (1 - alpha) * CrossEntropy(student, labels)

Test Set Results

The student outperforms the teacher across all metrics while being smaller and faster.

Dataset

• Dataset: Akash-Sakala/phishing-site-classification
• Train: 154,000 | Validation: 33,000 | Test: 33,000
• Labels: 0 = benign, 1 = phishing

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained('Akash-Sakala/bert-phishing-classifier_student')
model = AutoModelForSequenceClassification.from_pretrained('Akash-Sakala/bert-phishing-classifier_student')

url = 'http://suspicious-login.verify-account.com/secure'
inputs = tokenizer(url, return_tensors='pt', truncation=True, padding='max_length')

with torch.no_grad():
    logits = model(**inputs).logits

pred = torch.argmax(logits, dim=1).item()
print('Phishing' if pred == 1 else 'Benign')

Teacher Model

Akash-Sakala/bert-phishing-classifier_teacher