FINAL-Bench / ZTC-Judge-4B

ZTC-Judge-4B

Answer verification from a single forward pass, with zero generated tokens — at 4B.

ZTC-Judge-4B takes a question and an answer written by any model and scores whether that answer can be trusted. It is the bottom rung of a four-point size ladder measured under one identical protocol, and it is published so the shape of that ladder can be checked rather than asserted.

ZTC — Zero-Token Confidence · Judge — it evaluates someone else's answer, not its own

Read this before deploying

This model is not the strongest member of the family, and the card says so with numbers.

The ladder does not decline smoothly — it steps. Between 9B and 27B the score moves 0.078, while between 4B and 9B it moves 0.015. Whatever carries verification quality is largely absent below 27B on this axis.

Where this model is worth deploying is one specific place , and it is a real one:

🔴 Do not use this model for disaster and safety content. In that domain it does not clear the surface baseline, which means it is reading answer shape rather than correctness there.

✅ Professional-exam style content is where it earns its size. It runs on a laptop, on CPU, and inside networks that never reach the internet — places a hosted API cannot go.

How it works

[question + answer]  →  one forward pass
                     →  final-layer hidden state at the last position (2560-d)
                     →  probe
                     →  score

Generated tokens: 0. No access to the answering model's weights or logits is required; the text of the answer is the only input. Latency is one forward pass, and batching converts directly into throughput.

Usage

import json
import numpy as np, torch
from huggingface_hub import hf_hub_download, snapshot_download
from transformers import AutoModel, AutoTokenizer

REPO = "FINAL-Bench/ZTC-Judge-4B"
cfg  = json.load(open(hf_hub_download(REPO, "ztc_config.json"), encoding="utf-8"))
path = snapshot_download(REPO)

tok   = AutoTokenizer.from_pretrained(path)
model = AutoModel.from_pretrained(path, dtype=torch.bfloat16, low_cpu_mem_usage=True).eval()

def hidden(question, answer):
    b = tok([cfg["template"] % (question, answer)], return_tensors="pt",
            truncation=True, max_length=cfg["max_length"])
    dev = next(model.parameters()).device
    with torch.no_grad():
        h = model(input_ids=b["input_ids"].to(dev),
                  attention_mask=b["attention_mask"].to(dev)).last_hidden_state
    return h[0, int(b["attention_mask"].sum()) - 1].float().numpy().astype(np.float64)

# linear probe — one dot product
p = np.load(hf_hub_download(REPO, "ztc_probe.npz"))
v = hidden("Which defensive chemical does an insect release?", "C. Allomone")
print(float(((v - p["mu"]) / p["sd"]) @ p["w"]))

The score is an unbounded real number; higher means more likely correct. It is a ranking signal , not a calibrated probability — choose a threshold from your own review budget.

Two probes ship with this model

Both read the same input. The curved probe is the one to use when the number matters.

Protocol

The same protocol, item set and grading code are applied to every rung of the ladder and to the other systems on the independent leaderboard: https://huggingface.co/spaces/mayafree/typed-decision-leaderboard

Out of scope

• Not a grounding checker. It does not take a source document and decide whether the answer follows from it.
• Not a safety, toxicity or policy classifier.
• Not a calibrated probability. Use it to rank and threshold.
• Not a general-purpose verifier at this size. See the domain table above.

Limitations

• Domain coverage. Scores are meaningful only for the five domains listed. Outside them nothing has been measured and no guarantee is published.
• Below the surface baseline on disaster and safety. Stated in the table rather than omitted.
• Revision lock. The probe is fitted to one specific revision of the base model. Running it on a different revision produces no error and silently wrong scores ; this repository ships the matching weights so that failure mode cannot occur.
• It reports the verifier's judgement , which is not the answering model's own confidence — that quantity measures 0.5000 on this set.

Lineage

What this repository contains

The rest of the ladder

ZTC-Judge-27B · Darwin-397B-ZTC · ZTC-Judge-9B · ZTC-Judge-4B

License

The base model is Apache-2.0 and redistributable. The probes, the inference code and the evaluation tables are assets of FINAL-Bench / VIDRAFT.