The Latest AIs, every day
AIs with the most favorites on Toolify
AIs with the highest website traffic (monthly visits)
AI Tools by Apps
Discover the Discord of AI
AI Tools by browser extensions
GPTs from GPT Store
Discover The Best Model For AI
Top AI lists by month and monthly visits.
Top AI lists by category and monthly visits.
Top AI lists by region and monthly visits.
Top AI lists by source and monthly visits.
Top AI lists by revenue and real traffic.

Source model card:
Falconsai/proof_V_1@main, carried verbatim below. Its licence is the repository's. The Model Surgeon record follows it.
Falconsai/proof_V_1
)
A specialist–generalist decision model on a single DistilBERT encoder. The original 15-intent customer-support classifier is kept intact; a listwise cross-encoder head accepts any options you define at request time; and a step-function gate, tuned on validation data, decides which of the two answers each question.
falconsproof v1 extends
Falconsai/intent_classification
, pinned at revision
630d0d4668170a2a64d8d80b04d9844415bd4367
. It is the first version of the falconsproof line and the direct predecessor of falconsproof v2.0 (ModernBERT backbone, three specialists). This card describes
v1 only
.
Status of the numbers in this card. Every number comes from this checkpoint's
config.jsonandfalconsproof_report.json: one training run with thesmallpreset (1,500 training decisions per source, 1 epoch, seed 42). The tables between<!-- … -->markers are generated byfill_model_card.py; the analysis around them is written by hand. Only the environmental-impact section (§14) is still to be recorded.
| Task | Closed-set decisions: given a state (text), a question , and 2–20 options , return a calibrated probability per option |
| Backbone |
DistilBERT (6 layers, 768 hidden, 12 heads), initialised from
Falconsai/intent_classification
|
| Parameters | 67,556,476 in 109 tensors, float32 (as counted by Model Surgeon): encoder ≈ 66.4M; intent head ≈ 0.6M; option head ≈ 0.6M |
| Heads | Specialist : the original 15-intent head, frozen and preserved by distillation. Generalist : a listwise cross-encoder option head |
| Router |
Step-function gate
H(conf − τ) · H(coverage − κ) · H(map_conf − μ) · [no collisions]
, thresholds found by exhaustive search:
τ = 0.875, κ = 0.20, μ = 0.50
|
| Calibration |
One temperature per head, fitted by NLL on validation data:
t_intent
= 1.000,
t_option
= 0.709
|
| Context | 128 tokens (state and pairs are truncated beyond this) |
| Inference cost | Two encoder forward passes per call, however many questions are asked about the same state |
| Language | English |
| Output |
Probabilities, the chosen option, which head answered (
specialist
/
generalist
), and the gate's inputs
|
| Headline results (test) | Support routing (Bitext): 97.3% , above either head alone (specialist 95.0%, generalist 84.3%). All sources: 45.3% (specialist only 43.5%, generalist only 43.2%). Held-out Banking77: 45.0%. Multiple-choice QA: at or near chance |
The model has no generative component. It cannot produce text, only rank the options you give it.
| Field | Value |
|---|---|
| Model name | falconsproof v1 |
| Repository |
Falconsai/proof_V_1
|
| Developed by | Falconsai |
| Model type | Encoder-only decision model: classifier head plus cross-encoder ranking head with a thresholded router |
| Architecture class |
FalconsProofModel
(subclass of
DistilBertPreTrainedModel
),
model_type: "falconsproof"
|
| Config class |
FalconsProofConfig
(subclass of
DistilBertConfig
)
|
| Fine-tuned from |
Falconsai/intent_classification
@
630d0d4668170a2a64d8d80b04d9844415bd4367
|
| Language | English |
| License | Apache-2.0 for the model weights and code (verify against the base model's license before redistribution; see §7.4 for data licenses) |
| Custom code |
Yes:
falconsproof_modeling.py
ships with the weights. It is loaded with
importlib
, not with
trust_remote_code
|
| Training notebook |
falconsproof.ipynb
(v1)
|
| Intent labels (15) | cancellation, ordering, shipping, invoicing, billing and payment, returns and refunds, complaints and feedback, speak to person, edit account, delete account, delivery information, subscription, recover password, registration problems, appointment |
falconsproof v1 is intended for fast, local, closed-menu decisions over short English text , especially where one part of the traffic is customer-support intent routing and the rest is ad hoc.
| Use | Example |
|---|---|
| Support routing | State: an inbound message. Options: the 15 intents, or your own phrasings of them |
| Runtime-defined triage | State: a ticket. Options: your team names, written in plain language, changed at any time. Expect accuracy around 45% on unfamiliar taxonomies (CLINC150, Banking77), so use deferral (§13) |
| Short multiple-choice questions | Not recommended. Test accuracy on CommonsenseQA, OpenBookQA and ARC-Easy is at or near chance (§10.3) |
| Question fanout | Several questions (team, urgency, next step) about one message in one call |
| Selective automation | Act automatically on specialist-routed or high-confidence answers, and send the rest to a human |
The model can serve as a
routing or verification component
inside larger systems: an agent's tool-selection step, a pre-filter in front of an LLM, or a triage stage in a support pipeline. Its
route
output and calibrated confidence make a natural deferral signal (see §13).
Engineers who need sub-100 ms, deterministic, locally hosted decisions with interpretable routing, and who can evaluate the model on a sample of their own traffic before deploying.
falconsproof v1 is not suitable for:
pip install torch "transformers>=4.44,<5" safetensors huggingface_hub numpy
The model uses a custom class, which is stored in the repository and loaded from the downloaded snapshot. The repository holds
model_edited.safetensors
rather than
model.safetensors
and has no tokenizer files, so this loader handles both: it reads the weights under either name and falls back to the base model's tokenizer, which is the one v1 was trained with.
import importlib.util, json
import numpy as np, torch
from huggingface_hub import hf_hub_download, list_repo_files
from safetensors.torch import load_file
from transformers import AutoTokenizer
REPO = "Falconsai/proof_V_1"
BASE, BASE_REV = "Falconsai/intent_classification", "630d0d4668170a2a64d8d80b04d9844415bd4367"
files = list_repo_files(REPO)
# Model classes from the repo's modeling file
spec = importlib.util.spec_from_file_location("falconsproof_modeling", hf_hub_download(REPO, "falconsproof_modeling.py"))
fpm = importlib.util.module_from_spec(spec); spec.loader.exec_module(fpm)
# Config: use the repo's if complete, else the base DistilBERT config + falconsproof calibration and gate
raw = json.load(open(hf_hub_download(REPO, "config.json"), encoding="utf-8"))
report = (json.load(open(hf_hub_download(REPO, "falconsproof_report.json"), encoding="utf-8"))
if "falconsproof_report.json" in files else {})
config = (fpm.FalconsProofConfig.from_dict(raw) if {"dim", "n_layers", "id2label"} <= raw.keys()
else fpm.FalconsProofConfig.from_pretrained(BASE, revision=BASE_REV))
cal, gate = report.get("calibration", {}), report.get("gate", {})
for key, fallback in [("t_intent", cal.get("t_intent")), ("t_option", cal.get("t_option")), ("gate_tau", gate.get("tau")),
("gate_kappa", gate.get("kappa")), ("gate_map_min", gate.get("map_min")),
("option_template", None), ("max_len", None)]:
value = raw.get(key, fallback)
if value is not None:
setattr(config, key, value)
# Weights: accept model_edited.safetensors and strip any key prefix the editing tool added
weights_file = next((f for f in ("model.safetensors", "model_edited.safetensors") if f in files), None) \
or next(f for f in files if f.endswith(".safetensors"))
state = load_file(hf_hub_download(REPO, weights_file))
model = fpm.FalconsProofModel(config)
expected = set(model.state_dict())
strip = lambda k, p: k[len(p):] if k.startswith(p) else k
prefix = max(["", "model.", "module.", "falconsproof."], key=lambda p: len(expected & {strip(k, p) for k in state}))
missing, unexpected = model.load_state_dict({strip(k, prefix): v for k, v in state.items()}, strict=False)
missing = [k for k in missing if not k.endswith("position_ids")]
assert not missing, f"weights missing from {weights_file} (they would be random): {missing[:10]}"
if unexpected:
print("ignored unexpected tensors:", unexpected[:10])
model.eval() # add .to("cuda") for a GPU
# Tokenizer: the repo's if present, else the base model's (identical to what v1 was trained with)
try:
tokenizer = AutoTokenizer.from_pretrained(REPO)
except Exception:
tokenizer = AutoTokenizer.from_pretrained(BASE, revision=BASE_REV)
print("tokenizer: not in the repo, using the base model's")
cfg = model.config
LABELS = [cfg.id2label[i] for i in range(cfg.num_labels)]
print(f"loaded {weights_file}: {len(LABELS)} labels | tau={cfg.gate_tau:.3f} kappa={cfg.gate_kappa:.3f} "
f"mu={cfg.gate_map_min:.2f} t_intent={cfg.t_intent:.3f} t_option={cfg.t_option:.3f}")
# Expected for this checkpoint: tau=0.875 kappa=0.200 mu=0.50 t_intent=1.000 t_option=0.709
In v1, the routing logic lives in the training notebook rather than in
falconsproof_modeling.py
. This self-contained function reproduces it exactly:
def softmax(x, t=1.0):
x = np.asarray(x, dtype=np.float64) / max(float(t), 1e-6)
e = np.exp(x - x.max())
return e / e.sum()
@torch.no_grad()
def decide(state, questions):
"""questions: [{"question": str, "options": [str, ...]}, ...] -> one result per question (2 forward passes)."""
dev = model.device
# Pass 1: intent logits for the state and every unique option text (specialist).
singles = list(dict.fromkeys([state] + [o for q in questions for o in q["options"]]))
enc = tokenizer(singles, truncation=True, max_length=cfg.max_len, padding=True, return_tensors="pt")
il = model.intent_logits(enc["input_ids"].to(dev), enc["attention_mask"].to(dev)).float().cpu().numpy()
table = dict(zip(singles, il))
# Pass 2: one relevance logit per (question | state, option) pair (generalist).
firsts, seconds, spans = [], [], []
for q in questions:
first = cfg.option_template.format(question=q["question"], state=state)
spans.append((len(firsts), len(firsts) + len(q["options"])))
firsts += [first] * len(q["options"]); seconds += list(q["options"])
enc2 = tokenizer(firsts, seconds, truncation="longest_first", max_length=cfg.max_len, padding=True, return_tensors="pt")
scores = model.option_scores(enc2["input_ids"].to(dev), enc2["attention_mask"].to(dev)).float().cpu().numpy()
results = []
for q, (a, b) in zip(questions, spans):
generalist = softmax(scores[a:b], cfg.t_option)
native = softmax(table[state], cfg.t_intent)
maps, map_conf = [], []
for o in q["options"]:
p = softmax(table[o], cfg.t_intent)
maps.append(int(p.argmax())); map_conf.append(float(p.max()))
specialist = softmax(np.asarray(table[state])[maps], cfg.t_intent)
coverage = float(native[sorted(set(maps))].sum())
collision = len(set(maps)) < len(maps)
use_specialist = (not collision and min(map_conf) >= cfg.gate_map_min
and coverage >= cfg.gate_kappa and specialist.max() >= cfg.gate_tau)
probs = specialist if use_specialist else generalist
results.append({
"question": q["question"],
"choice": q["options"][int(np.argmax(probs))],
"confidence": float(probs.max()),
"route": "specialist" if use_specialist else "generalist",
"probs": {o: round(float(p), 4) for o, p in zip(q["options"], probs)},
"gate": {"conf": float(specialist.max()), "coverage": coverage, "min_map_conf": min(map_conf),
"collision": collision, "mapped": [LABELS[m] for m in maps]},
})
return results
for r in decide(
"I was charged twice for my March invoice, please reverse the duplicate payment.",
[{"question": "Which team should handle this message?", "options": ["billing and payment", "shipping", "delete account"]},
{"question": "How urgent is this?", "options": ["Low", "Medium", "High"]}],
):
print(r["question"], "->", r["choice"], f"({r['confidence']:.0%}, {r['route']})")
Tips for writing options. Options that are the exact intent names, or close paraphrases of them, give the specialist a confident mapping, which lets its gate open. Options that don't correspond to any of the 15 intents are handled by the generalist. Keep options distinct: two options that map to the same intent create a collision , and the specialist then abstains.
┌──────────────────── shared DistilBERT encoder (from Falconsai) ─────────────────────┐
state ─────────────┤─► [CLS] ─► intent head (15 labels, frozen) ─► specialist distribution over options │──┐
option texts ──────┤─► [CLS] ─► intent head ─► maps each option to an intent (+ mapping confidence) │ │
(question | state, │ │ │
option) pairs ───┤─► [CLS] ─► option head (MLP → 1 logit per pair) ─► generalist distribution │──┤
└──────────────────────────────────────────────────────────────────────────────────────┘ │
▼
step-function gate: specialist ⇔ H(conf − τ) · H(coverage − κ) · H(min_map_conf − μ) · [no collisions]
This is DistilBERT, initialised from the Falconsai checkpoint, and both heads read its
[CLS]
vector. During training the embeddings and the first two transformer layers are frozen, while the upper four layers adapt.
The
pre_classifier → ReLU → dropout → classifier
stack keeps exactly the parameter names of
DistilBertForSequenceClassification
, so the Falconsai weights load into it unchanged. It is frozen throughout training. At inference it does three jobs:
It also reports coverage : the share of the state's full 15-way probability that falls on the intents behind your options.
Each option is encoded as a pair
"{question} | {state}" [SEP] "{option}"
, and a two-layer MLP (
768 → 768 → GELU → dropout 0.2 → 1
) turns the pair's
[CLS]
vector into a relevance logit. A softmax over one question's options gives the distribution. Because the options are
inputs
rather than output classes, the label space is defined at request time.
| Step | Passes when | Guards against |
|---|---|---|
H(conf − τ)
|
The specialist's calibrated top probability is ≥ τ | Uncertain in-domain calls |
H(coverage − κ)
|
At least κ of the specialist's belief falls on the offered intents | Inputs outside the specialist's domain |
H(min_map_conf − μ)
|
Every option maps to an intent with confidence ≥ μ | Options the specialist can't interpret |
[no collisions]
|
Every option maps to a different intent | Options the specialist can't tell apart |
When all steps pass, the specialist answers; otherwise the generalist does. The thresholds are stored in
config.json
as
gate_tau
,
gate_kappa
and
gate_map_min
; the initial value
1.01
means closed.
In this checkpoint the gate is open, with τ = 0.875, κ = 0.20 and μ = 0.50
(§9).
| Decision | Reason |
|---|---|
| One shared encoder | One artifact to deploy, and the generalist starts from support-domain representations |
| Frozen head plus distillation | Freezing the head alone doesn't stop encoder drift. The distillation (KD) term keeps the specialist's outputs faithful to the teacher (the Learning-without-Forgetting pattern) |
| Cross-encoder, listwise | Joint attention over question, state and option, trained to produce a distribution over the options |
| Hard gate, searched rather than learned | Step functions have no useful gradient, and only three thresholds need choosing. Every answer then comes from one named head with an auditable reason |
| Domain-balanced objective | Stops the larger domain from dictating the thresholds |
| Coverage as a signal | Detects inputs the specialist believes are about something that wasn't offered, even when its restricted confidence is high |
Every example is converted to one schema:
{"state": "…", "question": "…", "options": ["…", "…"], "answer": 1,
"source": "clinc150", "domain": "general", "heldout": false, "intent_label": null}
Options are shuffled for every example, so the model can't learn a position shortcut.
| Source | Hugging Face id | Role | Splits used | How decisions are built |
|---|---|---|---|---|
| Bitext customer support |
bitext/Bitext-customer-support-llm-chatbot-training-dataset
|
In-domain (train, val, test) |
carved from
train
|
Bitext intents are mapped to Falconsai labels empirically (§7.3). Options are Falconsai labels, shown as label names or as natural paraphrases in equal measure |
CLINC150 (
plus
)
|
clinc/clinc_oos
|
General intents (train, val, test) | train / validation / test | 150 unseen intent names (plus "out of scope") as runtime-defined options |
| CommonsenseQA |
tau/commonsense_qa
|
General reasoning (train, val, test) | train (train and val carved) / validation → test | 5-way multiple choice |
OpenBookQA (
main
)
|
allenai/openbookqa
|
General reasoning (train, val, test) | train / validation / test | 4-way multiple choice |
| Banking77 |
PolyAI/banking77
|
Held out (test only) | test | Unseen intent label space |
| ARC-Easy |
allenai/ai2_arc
(
ARC-Easy
)
|
Held out (test only) | test | Unseen multiple-choice source |
For intent sources, each decision has 2–6 options: the gold intent plus random distractors from the same label space. If Bitext can't be downloaded, the notebook falls back to a small synthetic in-domain set built from label paraphrases, and says so in its output.
The first 64 examples of each Bitext intent are classified by the frozen Falconsai teacher. A Bitext intent is mapped to the teacher's majority label if at least
70%
of those examples agree, and dropped otherwise. The 64 mapping examples per intent are then excluded from the training, validation and test pools. The resulting mapping is stored in
falconsproof_report.json
under
bitext_intent_map
.
27 Bitext intents were mapped.
| Bitext intent | Falconsai label |
|---|---|
cancel_order
|
cancellation |
change_order
|
ordering |
change_shipping_address
|
shipping |
check_cancellation_fee
|
cancellation |
check_invoice
|
invoicing |
check_payment_methods
|
billing and payment |
check_refund_policy
|
returns and refunds |
complaint
|
complaints and feedback |
contact_customer_service
|
speak to person |
contact_human_agent
|
speak to person |
create_account
|
edit account |
delete_account
|
delete account |
delivery_options
|
delivery information |
delivery_period
|
delivery information |
edit_account
|
edit account |
get_invoice
|
invoicing |
get_refund
|
returns and refunds |
newsletter_subscription
|
subscription |
payment_issue
|
billing and payment |
place_order
|
ordering |
recover_password
|
recover password |
registration_problems
|
registration problems |
review
|
complaints and feedback |
set_up_shipping_address
|
shipping |
switch_account
|
edit account |
track_order
|
ordering |
track_refund
|
returns and refunds |
All 27 Bitext intents passed the 70% agreement threshold, so none were dropped. They cover
14 of the 15
Falconsai labels: Bitext has no appointment intent, so
appointment
is never tested in-domain. Some mappings reflect how the Falconsai classifier sees them rather than the Bitext names, for example
create_account → edit account
(not
registration problems
) and
track_order → ordering
(not
delivery information
).
Check each dataset's card before you redistribute derived data. At the time of writing, the listed licenses are:
| Dataset | License |
|---|---|
| Bitext customer support | CDLA-Sharing-1.0 |
| CLINC150 | CC BY 3.0 |
| CommonsenseQA | MIT |
| OpenBookQA | Apache-2.0 |
| Banking77 | CC BY 4.0 |
| ARC | CC BY-SA 4.0 |
Volumes are set by the training preset: at most
per_source_train
decisions per source for training, and
per_source_eval
for validation and for test.
| Preset | Train per source | Val / test per source | Epochs | Selected when |
|---|---|---|---|---|
smoke
|
250 | 80 | 1 | Pipeline check on a CPU |
small
|
1,500 | 300 | 1 |
auto
on a CPU
|
standard
|
4,000 | 500 | 2 |
auto
on a GPU
|
Actual counts per split and source for this checkpoint:
| Source | train | val | test |
|---|---|---|---|
| arc_easy | 0 | 0 | 300 |
| banking77 | 0 | 0 | 300 |
| bitext | 1500 | 300 | 300 |
| clinc150 | 1500 | 300 | 300 |
| commonsense_qa | 1485 | 297 | 296 |
| openbookqa | 1498 | 300 | 300 |
| Total | 5983 | 1197 | 1796 |
Totals: 5,983 training, 1,197 validation and 1,796 test decisions. The test split includes 600 held-out decisions (Banking77 and ARC-Easy). Multiple-choice sources lose a few rows to malformed items.
The student is built with
FalconsProofModel.from_pretrained("Falconsai/intent_classification", revision=…)
. A frozen copy of the original classifier is loaded as the
teacher
. Before training, the student's intent logits must match the teacher's to within
1e-4
, and the notebook stops if they don't.
L = CE_listwise(option logits of each decision, answer) + λ · T² · KL( softmax(teacher/T) ‖ softmax(student/T) )
The listwise cross-entropy is computed over each decision's options, with padding masked at
−1e4
. The distillation term uses the
states
of the training batch as inputs, with
λ = 1.0
and
T = 2.0
.
| Component | Trainable |
|---|---|
| Embeddings | No |
| Transformer layers 1–2 | No |
| Transformer layers 3–6 |
Yes (lr
3e-5
)
|
Intent head (
pre_classifier
,
classifier
)
|
No |
| Option head |
Yes (lr
5e-4
)
|
| Hyperparameter | Value |
|---|---|
| Optimiser | AdamW, weight decay 0.01 |
| Learning rate, encoder / option head | 3e-5 / 5e-4 |
| Schedule | Linear, 6% warmup |
| Batch | 16 decisions (flattened to all their option pairs) |
| Gradient clipping | 1.0 |
| Maximum sequence length | 128 |
| Options per decision | 2–6 when sampled, or all choices for multiple-choice sources |
| Distillation λ / T | 1.0 / 2.0 |
| Precision | bf16 autocast on CUDA when supported, fp32 otherwise |
| Seed | 42 |
| Checkpoint selection | Best generalist validation accuracy across epochs |
Monitored during training: listwise CE, distillation KL, generalist validation accuracy, and specialist agreement with the teacher on in-domain validation states.
After training, one temperature per head is fitted by minimising NLL on the validation set, using L-BFGS over
log T
and clamping the result to [0.05, 20].
t_option
is fitted on every validation decision;
t_intent
is fitted on the 15-way intent logits of in-domain validation states. Calibration changes confidence but never the argmax.
For each validation decision, the search computes the specialist's calibrated view (confidence, coverage, minimum mapping confidence, collisions) and the generalist's calibrated choice. It then evaluates every combination of:
| Threshold | Grid |
|---|---|
| τ (confidence) | 41 values, 1.00 → 0.00 |
| κ (coverage) | 21 values, 1.00 → 0.00 |
| μ (mapping confidence) | 0.9, 0.7, 0.5, 0.3, 0.0 |
That is 4,305 settings in all, vectorised in NumPy. The objective is domain-balanced accuracy , the mean of in-domain accuracy and general accuracy. Grids run from high to low, so ties go to the most conservative gate, the one that routes the fewest decisions to the specialist.
These values are read from this checkpoint's
config.json
and
falconsproof_report.json
.
| Parameter | Value |
|---|---|
Intent-head temperature
t_intent
|
1.000 |
Option-head temperature
t_option
|
0.709 |
Confidence step τ (
gate_tau
)
|
0.875 |
Coverage step κ (
gate_kappa
)
|
0.200 |
Mapping step μ (
gate_map_min
)
|
0.50 |
| Validation objective (domain-balanced accuracy) | 0.6784 |
| Validation decisions routed to the specialist | 31.4% |
Base model:
Falconsai/intent_classification@630d0d4668170a2a64d8d80b04d9844415bd4367
.
What these values mean in practice. The specialist answers a question only when four things hold. Its calibrated top probability is at least 87.5% . At least 20% of its belief falls on the intents behind the offered options. Every option maps to an intent with at least 50% confidence. And no two options share an intent. On validation data that happened for 31.4% of decisions, and the gated model reached 67.8% domain-balanced accuracy.
Temperatures.
t_option
= 0.709 is below 1, so the generalist's raw scores were under-confident and calibration sharpens them.
t_intent
is exactly 1.000, so the specialist's probabilities are the intent head's raw softmax. An exact 1.0 can also mean the fit never moved from its starting value. Since the specialist's ECE on Bitext is low (0.025, §10), this doesn't hurt in-domain, but it is worth re-checking when retraining.
Reading the values. A τ close to 1 means the specialist answers only when it is nearly certain. A κ close to 1 means it answers only when the options cover almost all of its belief. A gate value above 1 means that condition never passes, so the specialist is effectively disabled.
The test split contains all in-domain and general sources, plus the held-out sources Banking77 and ARC-Easy. Held-out data was never used for training, calibration or gate tuning. Metrics are reported per source and for all sources together:
| Metric | Definition |
|---|---|
| Specialist | Accuracy of the specialist path alone, ignoring the gate |
| Generalist | Accuracy of the generalist path alone |
| falconsproof | Accuracy of the gated model, the one you deploy |
| Balanced | Mean per-class recall of the gated model, where the class is the position of the correct option |
| Perturbed | Accuracy after the options are reshuffled (seed 49), which tests position robustness |
| ECE | Expected calibration error of the final probabilities (10 equal-width bins) |
| Route | Share of decisions answered by the specialist |
| Source | n | Specialist | Generalist | falconsproof | Balanced | Perturbed | ECE | Route → specialist |
|---|---|---|---|---|---|---|---|---|
| bitext | 300 | 0.950 | 0.843 | 0.973 | 0.975 | 0.973 | 0.025 | 87.7% |
| clinc150 | 300 | 0.453 | 0.450 | 0.463 | 0.414 | 0.463 | 0.118 | 16.3% |
| commonsense_qa | 296 | 0.226 | 0.250 | 0.250 | 0.249 | 0.250 | 0.068 | 5.7% |
| openbookqa | 300 | 0.253 | 0.280 | 0.267 | 0.267 | 0.267 | 0.110 | 6.7% |
| arc_easy (held out) | 300 | 0.243 | 0.317 | 0.313 | 0.449 | 0.313 | 0.089 | 5.0% |
| banking77 (held out) | 300 | 0.483 | 0.447 | 0.450 | 0.371 | 0.450 | 0.221 | 18.3% |
| All sources | 1796 | 0.435 | 0.432 | 0.453 | 0.462 | 0.453 | 0.087 | 23.3% |
Preset
small
, seed 42, max length 128, 1 epoch(s). Accuracies are fractions of 1.
| Source | Chance* | Best single head | falconsproof | Gain from gating | Specialist route |
|---|---|---|---|---|---|
| Bitext (in-domain) | ≈ 0.29 | 0.950 (specialist) | 0.973 | +2.3 pts | 87.7% |
| CLINC150 | ≈ 0.29 | 0.453 (specialist) | 0.463 | +1.0 pts | 16.3% |
| Banking77 (held out) | ≈ 0.29 | 0.483 (specialist) | 0.450 | −3.3 pts | 18.3% |
| CommonsenseQA | 0.20 | 0.250 (generalist) | 0.250 | 0.0 pts | 5.7% |
| OpenBookQA | 0.25 | 0.280 (generalist) | 0.267 | −1.3 pts | 6.7% |
| ARC-Easy (held out) | ≈ 0.25 | 0.317 (generalist) | 0.313 | −0.3 pts | 5.0% |
| All sources | 0.435 (specialist) | 0.453 | +1.8 pts | 23.3% |
*Chance is the expected accuracy of a random pick. Intent decisions have 2–6 options chosen uniformly at random (mean of 1/k ≈ 0.29); CommonsenseQA has 5 options and OpenBookQA 4.
tune_gate
).
small
preset (1 epoch, 1,500 decisions per source) and a 67M-parameter encoder were not enough to learn multiple-choice QA. Treat any use outside support routing as unproven.
Run
falconsproof.ipynb
top to bottom with
preset = "small"
and seed 42. The notebook writes the table above to
falconsproof_report.json
. Results vary with the preset, the hardware (bf16 versus fp32), and dataset versions on the Hub.
The training notebook runs these checks on the model after reloading it from disk . The notebook's printed output records whether each one passed for a given training run:
| # | Check | Pass criterion |
|---|---|---|
| 1 | Probabilities are valid | Each question's probabilities sum to 1 (±1e-6) |
| 2 | Determinism | Two identical calls return identical probabilities |
| 3 | Save/load fidelity | Option and intent logits match to <1e-4; temperatures, gate and template round-trip; test metrics identical after reload |
| 4 | Fanout consistency | Asking questions together equals asking them separately (±1e-3) |
| 5 | Order robustness | The chosen option text is unchanged after shuffling, on ≥ 90% of 200 test decisions |
| 6 | In-domain routing | "I forgot my password and can't sign in" with intent-name options routes to the specialist |
| 7 | Off-domain routing | A deployment question ("canary shows a 3% error increase") routes to the generalist |
| 8 | Specialist preserved | The intent head agrees with the original Falconsai model on ≥ 95% of in-domain test states |
A failing check is treated as a finding about the checkpoint, typically an undertrained preset, rather than as a crash.
Technical limitations
small
preset: multiple-choice accuracy on science and commonsense questions is at or near chance (25–31%).
Sociotechnical risks
standard
preset on a GPU
(4,000 decisions per source, 2 epochs) before relying on the generalist, and re-tune the gate on your own traffic if it resembles Banking77 more than Bitext.
Not yet filled. Record the hardware, training time and region of your training run here. Estimate emissions with the ML CO2 Impact calculator .
| Hardware | — |
| Training time | — |
| Cloud provider / region | — |
| Estimated emissions | — |
This checkpoint was trained with the
small
preset (about 6k training decisions, 1 epoch), which the notebook's
auto
setting selects on a CPU. As a guide, v1 is small: the
standard
preset runs about 10k decisions for 2 epochs on a 67M-parameter model, which takes minutes to tens of minutes on a single modern GPU. Inference needs no GPU.
| Objective | Listwise CE + KD (T = 2) |
| Pooling |
[CLS]
token of the final layer
|
| Pair format |
"{question} | {state}" [SEP] "{option}"
, truncated
longest_first
to 128 tokens
|
| Inference passes | 2 per call: (1) state and unique option texts through the intent head; (2) all (question, option) pairs through the option head |
| Software | Python ≥ 3.9 (3.11 recommended), PyTorch ≥ 2.1, transformers ≥ 4.44 and < 5, safetensors, NumPy. This checkpoint's config was saved with transformers 4.57.6 |
| Serialisation |
model_edited.safetensors
(float32, also as
model_edited.F32.gguf
) plus
config.json
(
model_type: falconsproof
,
architectures: ["FalconsProofModel"]
)
|
| Determinism | Bit-for-bit repeatable in eval mode with the same hardware and library versions |
Config fields added on top of
DistilBertConfig
:
| Field | Meaning | Initial | This checkpoint |
|---|---|---|---|
t_intent
|
Intent-head temperature | 1.0 | 1.000 |
t_option
|
Option-head temperature | 1.0 | 0.709 |
gate_tau
|
Confidence step τ | 1.01 (closed) | 0.875 |
gate_kappa
|
Coverage step κ | 1.01 (closed) | 0.200 |
gate_map_min
|
Mapping-confidence step μ | 1.01 (closed) | 0.50 |
option_template
|
First-segment template for pairs |
"{question} | {state}"
|
same |
max_len
|
Tokenizer truncation length | 128 | 128 |
The other DistilBERT settings in
config.json
are unchanged from the base model: 6 layers, 12 heads, dim 768, hidden dim 3072, vocabulary 30,522, 512 positions, dropout 0.1, and classifier dropout 0.2.
| File | Contents |
|---|---|
config.json
|
Architecture, 15 labels, temperatures, gate thresholds |
model_edited.safetensors
|
All weights (float32, 270 MB): encoder, intent head, option head |
model_edited.F32.gguf
|
The same weights in GGUF format |
falconsproof_modeling.py
|
FalconsProofConfig
and
FalconsProofModel
|
falconsproof_report.json
|
Base-model revision, labels, calibration, gate search, Bitext mapping, training config, data counts, test results |
lineage.intoto.jsonl
,
manifest.json
,
surgery_log.json
|
Model Surgeon lineage attestation, manifest and operation log |
falconsai_surgeon_package_*.zip
|
The complete Model Surgeon package |
README.md
|
This model card |
The repository has no
model.safetensors
and no tokenizer files, so plain
from_pretrained("Falconsai/proof_V_1")
fails. The loader in §5.2 handles both. The tokenizer is the base model's,
Falconsai/intent_classification
at the pinned revision.
| v1 (this model) | v2.0 | |
|---|---|---|
| Backbone | DistilBERT, 128 tokens | ModernBERT-base, trained up to 512 tokens |
| Specialists | 1 (the original Falconsai head, reused) | 3 (intent, code, reasoning), each distilled from its own teacher |
| Gate | One, with three thresholds | One per specialist, found by coordinate-ascent search |
| Training data | Support intents, general intents, multiple-choice QA | Adds 11 code tasks and 7 reasoning sources |
| Routing code | In the notebook (reproduced in §5.3) |
Shipped in
falconsproof_modeling.py
|
Changelog
small
preset (seed 42). Frozen Falconsai specialist with distillation, listwise cross-encoder generalist, temperature scaling, and a single step-function gate found by exhaustive search.
| Term | Meaning |
|---|---|
| State | The text the decision is about |
| Decision | One question with its options |
| Specialist | The original 15-intent classifier head |
| Generalist | The cross-encoder option head, which handles any options |
| Coverage | The share of the specialist's belief that falls on the intents behind the offered options |
| Mapping confidence | How sure the specialist is about which intent an option means |
| Collision | Two options mapped to the same intent |
| Fanout | Several questions about one state, answered in the same two forward passes |
| Held out | Never seen in training, calibration or gate tuning |
| ECE | Expected calibration error: the average gap between confidence and accuracy |
@misc{falconsai_proof_v1_2026,
title = {falconsproof v1: a specialist-generalist decision model with a step-function gate},
author = {{Falconsai}},
year = {2026},
howpublished = {\url{https://huggingface.co/Falconsai/proof_V_1}},
note = {Fine-tuned from Falconsai/intent_classification}
}
Methods
Datasets
Written by the Falconsai team. Please report issues, misroutes or evaluation results through the Community tab of this repository.
This card is generated from the surgical record itself; the package's
lineage.intoto.jsonl
is the signed source of truth (verify it free at
the Surgeon's public verifier or with the bundled
verify_attestation.py
).
safetensors
· Intended task: not declared
config.json
: the repo's config.json, edited
ed25519:70d5116dbd76
(FALCONS.AI Model Surgeon V7.96; 4 earlier operation(s) carried) · Falconsai/proof_V_1
load_and_test.py
.
The signed attestation + this card together document model composition, modification history, and validation evidence — the record structure technical-documentation obligations (e.g. EU AI Act Annex IV) ask for. This is evidence, not legal advice.
Operated with Model Surgeon — verify this package at https://surgeon.falcons.ai/verify © 2026 FALCONS.AI — Model Surgeon record format. The model weights remain their owner's.
proof_V_1 huggingface.co is an AI model on huggingface.co that provides proof_V_1's model effect (), which can be used instantly with this Falconsai proof_V_1 model. huggingface.co supports a free trial of the proof_V_1 model, and also provides paid use of the proof_V_1. Support call proof_V_1 model through api, including Node.js, Python, http.
proof_V_1 huggingface.co is an online trial and call api platform, which integrates proof_V_1's modeling effects, including api services, and provides a free online trial of proof_V_1, you can try proof_V_1 online for free by clicking the link below.
proof_V_1 is an open source model from GitHub that offers a free installation service, and any user can find proof_V_1 on GitHub to install. At the same time, huggingface.co provides the effect of proof_V_1 install, users can directly use proof_V_1 installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
