Total runs:
533.2K
Run Growth:
-256.3K
Growth Rate:
-47.93%
Sponsored by
Vmake Labs Video Enhancer
FINAL-Bench / ZTC-Judge-27B
Model's Last Updated: September 22 2026
text-classification
Introduction of ZTC-Judge-27B
Model Details of ZTC-Judge-27B
ZTC ecosystem
Zero-Token Confidence (ZTC)
reads a model's own internal state to judge whether an answer is right —
generating zero tokens
.
Darwin-397B-ZTC
ships a self-readout probe;
ZTC-Judge-27B
judges other models' answers; the arcade shows the gate deciding
execute or hold
in real time.
ZTC-Judge-27B
Ask a model how confident it is and you get a coin flip (AUC 0.500). This reads the same forward pass and gets 0.7282 on domains it has never seen — above the surface baseline in all five.
ZTC-Judge-27B takes a question and an answer written by any model and scores whether that answer can be trusted — without generating a single token .
ZTC — Zero-Token Confidence Judge — it evaluates someone else's answer, not its own
How it works
[question + answer] → one forward pass through the 27B model
→ final-layer hidden state at the last position (5,120-d)
→ one dot product with the probe
→ score
Generated tokens: 0. No access to the answering model's weights, logits or log-probabilities is required — the text of the answer is the only input. Because nothing is decoded, latency is one forward pass and batching translates directly into throughput.
Usage
from ztc_judge import ZTCJudge
judge = ZTCJudge.from_pretrained("FINAL-Bench/ZTC-Judge-27B")
judge.score(
"Which defensive compound does the insect release when it meets a predator?",
"C. Allomone",
domain="scientific_reasoning",
)
# {'score': -0.61, 'verdict': 'review', 'domain_auc': 0.7410, 'generated_tokens': 0}
Batched scoring (recommended — there is no decode step, so batch size is throughput):
judge.score_batch([(q1, a1), (q2, a2), ...], domain="scientific_reasoning")
The score is an unbounded real number; higher means more likely correct. It is a ranking signal , not a calibrated probability. Pick a threshold from your own review budget (see below).
Evaluation
Two independent measurements are reported. They use different test sets and different protocols, and are not interchangeable — read the protocol line before quoting a number.
A. Independent leaderboard — 2,018 items, leave-one-domain-out
The Typed Decision Leaderboard scores answer verifiers from several vendors on one identical item set with identical labels: https://huggingface.co/spaces/mayafree/typed-decision-leaderboard
| System | AUC |
|---|---|
| Darwin-397B-ZTC | 0.7364 |
| JEV (TypeSafe AI) | 0.7350 |
| ZTC-Judge-27B | 0.7282 |
| GPT-5.2 asked directly | 0.7148 |
| open-jev 4B | 0.6844 |
| Answer length and formatting only | 0.6223 |
| Patronus Lynx 8B | 0.5179 |
| The answering model's own stated confidence | 0.5000 |
Per domain, and against the surface baseline in the same domain:
| Domain | Baseline | ZTC-Judge-27B | Items | Wrong |
|---|---|---|---|---|
| Professional exams (law · math · biology) | 0.7138 | 0.8462 | 400 | 80 |
| Scientific reasoning | 0.7272 | 0.7410 | 198 | 20 |
| Biology & medicine | 0.5908 | 0.7154 | 917 | 169 |
| Disaster & safety procedures | 0.5949 | 0.6961 | 225 | 109 |
| General multi-step reasoning | 0.5420 | 0.6172 | 278 | 130 |
| Size-weighted mean | 0.6223 | 0.7282 | 2,018 | 508 |
Above the surface baseline in all five domains. That baseline — a model reading nothing but answer length, digit count and formatting — is the bar that matters; a verifier below it is detecting shape, not correctness.
Protocol. Every figure comes from a domain the probe never saw. Hyper-parameters are selected inside the training domains only. Scores are computed per domain and then size-weighted; pooling all items into one AUC inflates the result, because score scales differ between domains.
Self-readout. Given only the question, this model answers on its own and the same forward pass tells whether it was right: 0.7322 (3 domains, 1,595 items). Tools that see only text from outside a model cannot do this at all.
B. Repository-internal measurement — 400 items, random folds
| Judge |
Qwen3.5-27B, revision
fc05daec18b0
(weights unmodified)
|
| Items | 400 professional-exam items, out-of-fold |
| Result | 0.7999 |
| Permutation null | z = 25.68 |
This number is higher than the leaderboard figure because the protocol is easier: one domain, random folds instead of held-out domains. Quote 0.7282 when comparing against other systems.
Two probes ship with this model
| File | Size | Produces |
|---|---|---|
ztc_probe.npz
|
59 KB | linear readout — one dot product |
ztc_curve_probe.npz
|
5.3 MB | the leaderboard figure, 0.7282 — 256 anchors, RBF kernel |
usage_curve.py
|
— | runnable example for both |
Both read the same input: the final-layer hidden state at the last position, from a single forward pass with zero generated tokens. The curved probe is the one to use when the number matters; the linear probe is there for deployments where 59 KB and a single dot product are worth the difference.
Review budget
If you can only re-check part of your traffic, sort by score ascending and review the lowest.
| Reviewed | Errors caught | vs. random |
|---|---|---|
| 5% | 14.0% | 2.8× |
| 10% | 28.0% | 2.8× |
| 15% | 38.4% | 2.6× |
| 20% | 47.2% | 2.4× |
| 30% | 62.0% | 2.1× |
| 40% | 72.6% | 1.8× |
Intended use
- Triage of LLM output at scale — route low-scoring answers to a human, a larger model, or a retrieval step, and let the rest through.
- Regression monitoring — track the score distribution of a deployed system over time.
- Cheap pre-filter in front of an expensive verifier — costs one forward pass, no decoding.
Out of scope
- Not a grounding checker. It does not take a source document and decide whether the answer follows from it. Tools built for that task target a different axis.
- Not a safety, toxicity or policy classifier.
- Not a calibrated probability. Use it to rank and threshold, not as a likelihood.
- Not a substitute for human review in high-stakes decisions.
Limitations
-
Domain coverage.
Scores are only meaningful for the five domains above. When
domain_aucreturnsNone, the domain was never measured and the score carries no published guarantee. - Weakest domain is 0.6172 (general multi-step reasoning). General multi-step reasoning is the hardest case and is listed first in this section on purpose.
- Revision lock. The probe is fitted to one specific revision of the base model. Running it on a different Qwen3.5-27B revision produces no error and silently wrong scores . This repository ships the matching weights so that failure mode cannot occur.
- It reports the verifier's judgement , which is not the same thing as the answering model's own confidence — that quantity measures 0.500 here.
- Long answers are read within the model's context window; answers exceeding it are truncated before scoring.
Lineage
| Base model |
Qwen/Qwen3.5-27B, revision
fc05daec18b0
, Apache-2.0
|
| Modification to base weights | none — the probe is a separate 59 KB file |
| Added by FINAL-Bench | probe, inference code, evaluation protocol and tables |
What this repository contains
| Included | Base weights · tokenizer · probe · inference code · evaluation configuration |
| Not included | Training corpus · hidden-state matrices · fitting pipeline |
The probe is the product; the material it was fitted from is not distributed.
Independent comparison
This model is listed on the Typed Decision Leaderboard , which measures answer verifiers from several vendors on one identical item set with identical labels, and publishes scores, labels and grading code:
https://huggingface.co/spaces/mayafree/typed-decision-leaderboard
License
The base model, Qwen3.5-27B, is Apache-2.0 and redistributable. The probe, the inference code and the evaluation tables are assets of FINAL-Bench / VIDRAFT.
Citation
@misc{ztc_judge_27b_2026,
title = {ZTC-Judge-27B: zero-token answer verification from a single forward pass},
author = {FINAL-Bench},
year = {2026},
url = {https://huggingface.co/FINAL-Bench/ZTC-Judge-27B}
}
Runs of FINAL-Bench ZTC-Judge-27B on huggingface.co
471
Total runs
0
24-hour runs
9
3-day runs
52
7-day runs
465
30-day runs
More Information About ZTC-Judge-27B huggingface.co Model
Url of ZTC-Judge-27B
ZTC-Judge-27B huggingface.co Url
Provider of ZTC-Judge-27B huggingface.co
Other API from FINAL-Bench
Updated:September 25 2026
Total runs:
271.4K
Run Growth:
-61.7K
Growth Rate:
-22.74%
Total runs:
57.3K
Run Growth:
52.3K
Growth Rate:
100.00%
Total runs:
3.5K
Run Growth:
3.4K
Growth Rate:
97.95%
Total runs:
2.5K
Run Growth:
2.5K
Growth Rate:
100.00%
Total runs:
2.2K
Run Growth:
1.9K
Growth Rate:
86.73%
Total runs:
1.5K
Run Growth:
1.4K
Growth Rate:
97.81%
Total runs:
1.3K
Run Growth:
1.3K
Growth Rate:
100.00%
Total runs:
995
Run Growth:
605
Growth Rate:
60.80%
Total runs:
976
Run Growth:
976
Growth Rate:
100.00%
Total runs:
973
Run Growth:
973
Growth Rate:
100.00%
Total runs:
968
Run Growth:
-65
Growth Rate:
-6.61%
Total runs:
912
Run Growth:
497
Growth Rate:
54.50%
Total runs:
806
Run Growth:
535
Growth Rate:
66.38%
Total runs:
600
Run Growth:
-122
Growth Rate:
-20.33%
Total runs:
491
Run Growth:
322
Growth Rate:
65.58%
Total runs:
432
Run Growth:
432
Growth Rate:
100.00%
Total runs:
401
Run Growth:
239
Growth Rate:
59.60%
Total runs:
385
Run Growth:
49
Growth Rate:
12.73%
Total runs:
376
Run Growth:
376
Growth Rate:
100.00%
Total runs:
346
Run Growth:
260
Growth Rate:
75.14%
Total runs:
336
Run Growth:
314
Growth Rate:
93.45%
Total runs:
327
Run Growth:
131
Growth Rate:
39.70%
Total runs:
305
Run Growth:
60
Growth Rate:
19.67%
Total runs:
272
Run Growth:
-34
Growth Rate:
-12.32%
Total runs:
194
Run Growth:
194
Growth Rate:
100.00%
Total runs:
191
Run Growth:
8
Growth Rate:
4.15%
Total runs:
191
Run Growth:
191
Growth Rate:
100.00%
Total runs:
176
Run Growth:
-26
Growth Rate:
-14.36%
Total runs:
162
Run Growth:
89
Growth Rate:
54.94%
Total runs:
153
Run Growth:
-82
Growth Rate:
-53.59%
Total runs:
149
Run Growth:
0
Growth Rate:
0.00%
Total runs:
130
Run Growth:
54
Growth Rate:
41.54%
Total runs:
127
Run Growth:
114
Growth Rate:
89.76%
Total runs:
119
Run Growth:
77
Growth Rate:
64.71%
Total runs:
96
Run Growth:
96
Growth Rate:
100.00%
Total runs:
72
Run Growth:
-180
Growth Rate:
-250.00%
Total runs:
63
Run Growth:
37
Growth Rate:
58.73%
Total runs:
61
Run Growth:
61
Growth Rate:
100.00%
Total runs:
60
Run Growth:
60
Growth Rate:
100.00%
Total runs:
60
Run Growth:
17
Growth Rate:
28.33%
Total runs:
52
Run Growth:
52
Growth Rate:
100.00%
Total runs:
52
Run Growth:
52
Growth Rate:
100.00%
Total runs:
51
Run Growth:
51
Growth Rate:
100.00%
Total runs:
50
Run Growth:
50
Growth Rate:
100.00%
Total runs:
49
Run Growth:
34
Growth Rate:
69.39%
Total runs:
48
Run Growth:
48
Growth Rate:
100.00%
Total runs:
47
Run Growth:
16
Growth Rate:
34.04%
Total runs:
46
Run Growth:
46
Growth Rate:
100.00%
Total runs:
46
Run Growth:
46
Growth Rate:
100.00%
Total runs:
43
Run Growth:
43
Growth Rate:
100.00%
Total runs:
40
Run Growth:
40
Growth Rate:
100.00%
Total runs:
31
Run Growth:
22
Growth Rate:
70.97%
Total runs:
13
Run Growth:
6
Growth Rate:
46.15%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%