cnuland / llm-d-sc-sensitivity

llm-d-sc-sensitivity

A 384-dimensional sentence embedding model fine-tuned to separate data sensitivity tiers , for use as a routing signal by llm-d-sc .

The model does not emit a class directly. It produces an embedding that is ranked against labelled anchors ( anchors.json , shipped here), so the taxonomy stays as replaceable data rather than a frozen classification head.

Taxonomy

Intended use

Deciding whether a prompt may be served by an external model endpoint. NEVER_EGRESS and REGULATED prompts typically must stay on in-cluster models. The classifier emits ranked evidence only; enforcement is the caller's responsibility.

This is a routing signal, not a security control. It is a similarity ranking with a measured error rate. Do not use it as the sole barrier preventing secret exfiltration.

Evaluation

Evaluated by llm-d-sc on a held-out set of 75 prompts authored independently of the training corpus , 15 per tier, of which 25 are deliberate boundary cases (for example "What makes a password strong, in general terms?" is PUBLIC , not NEVER_EGRESS ).

Method: cosine similarity against the anchors, mean of the top 3 per tier, argmax.

Per tier:

NEVER_EGRESS separates perfectly: credential-bearing text is lexically and semantically distinctive. Residual error concentrates on the INTERNAL / CONFIDENTIAL boundary, which is genuinely a matter of organizational policy rather than of language.

Latency on CPU (single thread, Apple M-series, embed plus rank): p50 8.1 ms, p99 12.9 ms.

These numbers were produced on a homelab and have not been independently reproduced.

Known limitation: training data quality

The synthetic data pipeline that produced this model captured the generator's reasoning traces rather than the intended generated texts. Training sentences average 175 tokens and read like "The user wants a NEW example text that belongs to the PUBLIC sensitivity tier..." instead of like real user prompts.

The model still performs well above the base model, because anchor ranking depends on the geometry of the embedding space rather than on the training text distribution. But the gap between this model (0.8933) and the equivalently trained complexity model (0.9750, clean training data) is the likely cost of that defect. Retraining on corrected data is expected to close it.

Training

Fine-tuned from sentence-transformers/all-MiniLM-L6-v2 with BatchAllTripletLoss and group_by_label batch sampling: 495 synthetic examples, 20 epochs, batch size 16, lr 2e-5. Pipeline: https://github.com/cnuland/hello-chris-sr-finetuned

Limitations

• English only.
• Trained on synthetic data with the defect described above; no human-labelled validation set exists.
• Anchor quality directly determines accuracy. Replacing anchors.json changes behaviour without retraining.
• Sensitivity is organization-specific. The shipped anchors encode one reasonable policy, not yours.

License

Apache-2.0.