cnuland / llm-d-sc-route-gate

llm-d-sc-route-gate

route classifier for llm-d semantic classification . Labels: LARGE , SMALL .

Architecture: sequence-classification head (requires a runtime that reads logits) , base sentence-transformers/all-MiniLM-L6-v2 .

Accuracy

Read the real-traffic row first.

Run-to-run variance. PUBLISHED SEED IS THE FIRST ONE RUN (11), not the best. Both seeds: seed 11 refined gold 0.9335 real gold 0.9330 seed 22 refined gold 0.9202 real gold 0.9187 median 0.9269

WHAT THIS MODEL IS. The backend routing decision the deployed llm-d-sc/Praxis table actually makes: SIMPLE/MEDIUM -> the small model, COMPLEX/REASONING -> the large one. It is not a four-tier complexity classifier. A tier classifier saying MEDIUM where the jury said SIMPLE picks the same backend and has made no routing error; tier-exact accuracy charges it for one anyway. Folding a 4-way model's output onto this decision scores 88.22%; training it directly scores 93.35%.

THIS DECISION FIRES ON EVERY REQUEST, which is why it is worth a dedicated model even though its accuracy is the lowest of the four gates in this family.

READ THE BASELINE AND THE MINORITY CLASS. Majority baseline on refined gold is 80.59% (always answer SMALL). 93.35% clears it by 12.8 points, and LARGE recall is 91.78% on a class that is 19.4% of the eval -- the minority class is doing real work. LARGE precision is 77.91%, i.e. about 6% of small requests are over-routed to the large model. That is a cost trade, and the direction most deployments prefer to the reverse; move the threshold if yours does not.

WEAKEST POINT, STATED. On the 176 rows where the three-model jury SPLIT, it scores 77.84% against a 72.16% baseline -- clear of chance but 15 points below its clean-subset figure. Contested rows are about 30% of real traffic here.

The eval has a measured ceiling

Gold labels were audited by blind paired adjudication in two strata — the rows this model got wrong, and a sample of the rows it got right — with the judge shown two candidate labels in random order and no indication of provenance. Roughly 4.9% of the gold labels are themselves wrong , so a PERFECT classifier scored against this eval would reach about 0.95, not 1.0 .

Read the real-traffic accuracy against that ceiling, not against 100%. Auditing only a model's mistakes would move the number up artificially; sampling the correct rows too is what makes the estimate honest, and it revealed that on ~3.3% of "correct" rows the model agreed with a bad label — meaning measured accuracy is very slightly overstated.

How the eval was built

Real-traffic rows come from WildChat-1M (ungated real assistant traffic). Each prompt was labelled independently by three models ( claude-opus-5 , claude-sonnet-5 , claude-fable-5-1 ) from the task rubric alone -- no labeller ever saw a proposed label , so agreement is evidence rather than assent. Only unanimous rows are scored.

Those three agree unanimously on roughly 70-74% of real prompts. The remaining prompts are published as a contested split rather than discarded: they measure how much real traffic this taxonomy does not resolve, which no single accuracy figure can express.

Training data

418 rows from route-v2+route-real+route-active+route-distill+route-real-contested , mixing jury-labelled real traffic (register and class prior) with rubric-grounded synthetic data (coverage of tiers that are rare in real traffic). Training prior: None . Held-out eval prompts are excluded by content hash.

Latency

CPU single-request: p50 7.24 ms, p99 19.12 ms (Apple M-series, single thread). llm-d-sc serves the classifier on CPU, so model size trades directly against per-replica throughput.

Limitations

• WildChat is consumer traffic. For sensitivity it is ~93% PUBLIC and cannot measure the tiers that gate egress; the enterprise row above covers those.
• Labels come from LLM jurors, not human annotators. The rubric was validated by reproducing the project's hand-authored gold labels (complexity 0.9875, cost 1.000, sensitivity 1.000) before use.
• Not independently reproduced.