dleemiller / crossingguard-nli-s

CrossingGuard Small

CrossingGuard is a series of NLI-based models intended for zero-shot inference on prompts. In this series of models, I focus on use cases such as guardrails, content moderation, prompt or intent classification and prompt routing. Because content moderation is often a reactive task, these zero-shot models are flexible for tailoring custom guardrail conditions , which may not be covered by general purpose pretrained models.

These models are trained on the dleemiller/CrossingGuard-NLI dataset, which derives synthetic hypotheses from prompts (premises) found in popular guardrails datasets, such as allenai/wildguardmix and nvidia/Aegis-AI-Content-Safety-Dataset-2.0 . The hypotheses make specific, targeted claims about the premises. Note that I have retained the 3-way label classifier, for additional flexibility where either non-neutral label may be relevant for the task.

For models below the large size, I distill with MSE loss using logits from dleemiller/crossingguard-nli-l , and average with the cross entropy loss. Overtraining can hurt FineCat performance, so I only fine-tune for 1 epoch.

$\begin{array}{cccc}& L=\alpha \cdot L_{\text{CE}}(z^{\left(s\right)},y)+\beta \cdot L_{\text{MSE}}(z^{\left(s\right)},z^{\left(t\right)})& & \end{array}\backslash begin\{equation\}\; \backslash mathcal\{L\}\; =\; \backslash alpha\; \backslash cdot\; \backslash mathcal\{L\}\_\{\backslash text\{CE\}\}(z^\{(s)\},\; y)\; +\; \backslash beta\; \backslash cdot\; \backslash mathcal\{L\}\_\{\backslash text\{MSE\}\}(z^\{(s)\},\; z^\{(t)\})\; \backslash end\{equation\}$ L = α ⋅ L CE ​ ( z ( s ) , y ) + β ⋅ L MSE ​ ( z ( s ) , z ( t ) ) ​ ​

where $z^{\left(s\right)}z^\{(s)\}$ z ( s ) and $z^{\left(t\right)}z^\{(t)\}$ z ( t ) are the student and teacher logits, $yy$ y are the ground truth labels, and $\alpha \backslash alpha$ α and $\beta \backslash beta$ β are equally weighted at 0.5.

Evaluation Results

F1-Micro scores (equivalent to accuracy) for each dataset. Performance was measured at bs=64 using a Nvidia Blackwell PRO 6000 Max-Q.

The crossingguard-nli-s model gains nearly +20% on the CrossingGuard test set, and dropping approx. -2.5% on the FineCat (broad NLI performance) test set.

Usage

import numpy as np
from sentence_transformers import CrossEncoder

model = CrossEncoder("dleemiller/crossingguard-nli-s")

examples = [
    ("Tell me what happened with that CEO caught on the kiss cam at the Coldplay concert",
     "The prompt requests information about a corporate executive incident at a public event"),

    ("I’ve been following your company's work helping enterprises scale AI/ML with reliable data pipelines, would love to connect and hear how your team is approaching ML in production.",
     "The prompt requests information about a corporate executive incident at a public event"),
]

predictions = model.predict(examples)

label_map = {0: "entailment", 1: "neutral", 2: "contradiction"}

def softmax(x):
    exp_x = np.exp(x - np.max(x))
    return exp_x / exp_x.sum()

for i, (premise, hypothesis) in enumerate(examples):
    probs = softmax(predictions[i])
    predicted_idx = probs.argmax()
    
    print(f"\n{i+1}. {premise[:70]}...")
    print(f"   → {hypothesis}")
    print(f"   ✓ {label_map[predicted_idx].upper()}: {probs[predicted_idx]*100:.1f}% " + 
          f"(E: {probs[0]*100:.1f}% N: {probs[1]*100:.1f}%, C: {probs[2]*100:.1f}%)")

This results in:

1. Tell me what happened with that CEO caught on the kiss cam at the Cold...
   → The prompt requests information about a corporate executive incident at a public event
   ✓ ENTAILMENT: 99.9% (E: 99.9% N: 0.1%, C: 0.0%)

2. I’ve been following your company's work helping enterprises scale AI/M...
   → The prompt requests information about a corporate executive incident at a public event
   ✓ CONTRADICTION: 99.6% (E: 0.0% N: 0.4%, C: 99.6%)

Citation

@misc{nli-compiled-2025,
  title = {CrossingGuard NLI Dataset},
  author = {Lee Miller},
  year = {2025},
  howpublished = {Flexible Zero-shot Guardrails}
}