Sponsored by APIMart
Toolify Toolify logo
AI Models
plumbline-0.6b

dsup2026 / plumbline-0.6b

Run on huggingface.co
huggingface.co
Total runs: 216
24-hour runs: 10
7-day runs: 84
30-day runs: 216
Model's Last Updated: October 13 2026
zero-shot-classification

Introduction of plumbline-0.6b

Model Details of plumbline-0.6b

Plumbline-0.6B

Loosely inspired by Jev. Running on a CPU. Give it any text and any rule in plain English. It returns a score for this text breaks the rule . A 0.6B LoRA adapter over Qwen/Qwen3-Reranker-0.6B .

Treat it as a ranker for a review queue rather than an auto-decider. The numbers say why, and so does the section on what it cannot do.

The contribution

The interesting part is not the architecture or the size. It is the supervision.

  1. Every training label was machine-verifiable , like RLVR without the RL, so carrying no annotator noise and no ceiling from a teacher model's beliefs.

  2. The training set was designed to be orthogonal to the evaluation sets used here, now or in the future. It was held out by kind rather than by sampling: there was not one semantic rule in the training data. The adapter has never seen a rule about tone, disclosure, blame, evidence, sourcing or professional conduct.

Results

GOLDSET_RECIPE.md sets out how the conduct set was built.

The LegalBench rows are the seven contract_nli tasks, which ask whether a clause permits or forbids something under an NDA. They are the only slice here with expert human labels.

conduct rules, Claude-authored (160 rules, 1,920 items) AUC 95% CI F1 @optimal gap
Plumbline-0.6B 0.901 [0.877, 0.924] 0.931 0.147
Qwen3-Reranker-4B 0.841 [0.807, 0.874] 0.902 0.212
Qwen3-Reranker-0.6B (base) 0.826 [0.791, 0.860] 0.884 0.207
LegalBench contract clauses (14 rules, 1,927 items) AUC 95% CI F1 @optimal gap
Plumbline-0.6B 0.871 [0.831, 0.910] 0.863 0.043
Qwen3-Reranker-4B 0.935 [0.910, 0.957] 0.918 0.182
Qwen3-Reranker-0.6B (base) 0.824 [0.775, 0.873] 0.838 0.079

F1 @optimal fits a cutpoint per rule on that rule's own labelled items. It is an in-sample ceiling, unreachable on fresh data, and it inflates as cells shrink, so compare it across models on one slice and never across slices of different sizes. gap is how much of that a model gives up when one shared threshold is used for every rule instead.

A note on AUC, because it is the number most likely to mislead you here. It is threshold-free, which is both the point of it and the trap. AUC is computed inside each rule and then averaged, so it measures how well a model orders items within a rule and says nothing about whether two rules' scores sit on the same scale. A model can therefore order every rule perfectly and still have no single cutoff that works across them. It is also insensitive to prevalence, so a high AUC tells you nothing about what you get at a 5% violation rate. The LegalBench table is the concrete case: the 4B ranks better and is harder to use. Read AUC as ranking quality and gap as usability, and do not let the first stand in for the second.

What the numbers say

The training carried over to rules it never saw. Give each rule its own best cutoff and the adapter scores +0.047 F1 above the plain base model on conduct rules, and +0.025 above it on LegalBench . Nothing in the training set was about tone, blame or sourcing, so the model is applying something it picked up elsewhere rather than recalling something it was taught.

It beats a model seven times its size at a single operating point, including where it ranks worse. On LegalBench the 4B ranks better (AUC 0.935 against 0.871) yet gives up far more to a shared threshold (gap 0.182 against 0.043), because its scores do not sit where one cutoff can use them. Ranking well and being usable at a fixed threshold are different properties, and only the second is what a deployment gets.

Its scores are the most comparable across rules of the three models. The gap column is smallest for Plumbline on both semantic slices: 0.043 against 0.182 on LegalBench, and 0.147 against 0.212 on conduct rules. One constant needs that property in order to work at all, and it is the clearest thing the training bought.

Use it 
import re, torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

BASE = "Qwen/Qwen3-Reranker-0.6B"
tok = AutoTokenizer.from_pretrained(BASE)
model = PeftModel.from_pretrained(
    AutoModelForCausalLM.from_pretrained(BASE), "dsup2026/plumbline-0.6b").eval()
YES, NO = tok.convert_tokens_to_ids("yes"), tok.convert_tokens_to_ids("no")

PREFIX = ('<|im_start|>system\nJudge whether the Document meets the requirements based on '
          'the Query and the Instruct provided. Note that the answer can only be "yes" or '
          '"no".<|im_end|>\n<|im_start|>user\n')
SUFFIX = '<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'
PROHIBITION = re.compile(r"\b(must not|may not|shall not|must never|should not|cannot|"
                         r"do not|does not|never)\b", re.I)

def fail_logit(text, rule, item_noun="text"):
    body = (f"<Instruct>: Decide whether the {item_noun} satisfies the criterion.\n"
            f"<Query>: {rule}\n<Document>: {text}")
    ids = torch.tensor([tok.encode(PREFIX, add_special_tokens=False)
                        + tok.encode(body, add_special_tokens=False)[:300]
                        + tok.encode(SUFFIX, add_special_tokens=False)])
    with torch.no_grad():
        lg = model(input_ids=ids).logits[0, -1]
    raw = (lg[YES] - lg[NO]).item()
    # "yes" means the text MEETS the criterion. Flipping this sign costs 0.901 AUC -> 0.099.
    return raw if PROHIBITION.search(rule) else -raw

print(fail_logit("Call me on 052-000-0000 tonight.",
                 "The message must not contain a personal phone number."))

Higher means more likely to break the rule. Rank with it, and read the threshold section before comparing it to a constant.

Read this before anything else: polarity

yes means the text meets the criterion. For a prohibition, meeting it means the text fails . The sign has to flip depending on how the rule is phrased.

Getting this backwards turns 0.901 AUC into 0.099 , the same model on the same data, worse than chance, with no error message. It is the single largest correctness trap in using this model. The regex above is only a heuristic: a rule phrased as a description of a violation rather than as a rule will fool it. If your rules are not phrased with must or must not , check the sign against a handful of labelled examples first.

Only the adapter is published

This repository holds the LoRA delta and nothing else, which keeps the licence boundary clean. Every byte here is ours, and the base model stays Apache-2.0 from its own authors. merge.py merges it for you.

One measured caveat if you do merge. Applying a LoRA and merging it are mathematically identical but numerically different: the adapter path applies the low-rank term to ACTIVATIONS, while merging folds it into the WEIGHTS and rounds there. On this adapter that is worth up to 0.125 in the fail-logit and 2.6e-3 in probability , with every decision and the whole ordering unchanged. The adapter path is bit-deterministic across reloads, so the difference comes from the merge rather than from noise.

Every number in this card was measured on the adapter path. If you need them reproduced exactly, use the adapter directly rather than a merge.

What it cannot do

Conditional rules are much the weakest shape. Rules shaped may X only if Y , unless , or must not X without Y score AUC 0.716 , against 0.978 on plain requirements and 0.932 on prohibitions. Three of the forty measured fall below chance outright.

The failure has a specific shape, and how much it costs you depends on your traffic. Naming the exempting condition makes a text look more like a violation while making it compliant, so relevance and adherence point in opposite directions. Worse, a text that never triggers the condition at all complies vacuously, and the model reads that as a violation: measured against vacuously compliant texts alone, ranking inverts almost completely. If your rules are conditional and most of your documents never trigger the condition, expect far worse than 0.716. No threshold repairs a ranking failure. Split such a rule into two plain rules and combine the results yourself.

There is no single threshold that works. The gap column above is this limitation in aggregate. Here it is at its worst: across twelve semantic rules on one corpus of LinkedIn comments, the optimal cutoff ranged from 0.022 to 0.989 . On one of them the model ranks well (AUC 0.844) while F1 at a 0.5 threshold is 0.000 . Good ranking, catastrophic threshold.

So fit a threshold per rule on a few labelled examples of your own. Roughly eight is enough to recover about half of what a perfect threshold would give. A label-free method for this remains an open problem here: seven approaches were tried, and the best earned +0.013 F1 with a confidence interval spanning zero on rules it had not seen.

It is not a fact checker. On ANLI it scores below chance. It judges what a text does, not whether a claim is true.

It is not calibrated to your base rate. At 5% prevalence, catching 80% of violations leaves precision between 0.11 and 0.24.

Provenance

No Claude, GPT or other proprietary model contributed any training text or any label to training. Qwen was partially used.

The evaluation is a different matter, and the distinction is easy to miss. The conduct rules this card reports on were written by Claude. That material never enters training and is quarantined to evaluation, and Claude shares no lineage with the models under test, so it favours none of them. It is still not a human annotation. Every pair records its own origin, and the sets that are human-annotated say so: LegalBench carries expert annotation, and the machine-verifiable sets have no annotator and no teacher model anywhere in the labelling path.

Licence

PolyForm Noncommercial 1.0.0. Commercial use needs a separate licence.

The adapter is licensed under the PolyForm Noncommercial 1.0.0 , reproduced in full in LICENSE . Any noncommercial purpose is permitted with no permission needed, which its own text spells out as personal study, private entertainment, hobby projects and amateur pursuits, and which covers charities, schools, public research bodies and government whatever their funding.

Commercial use needs a separate licence, granted either by naming you in GRANTS.txt or in writing to you. GRANTS.txt is not a complete list. Running the model over a group or a queue that belongs to a business, a client or an employer is commercial use however informally it is run and whoever owns the laptop. Ask.

The base model, Qwen/Qwen3-Reranker-0.6B , remains Apache-2.0 and is not relicensed by this release; retain its LICENSE and NOTICE. This repository contains only the LoRA delta, so merging is yours to do and the base terms continue to govern the base weights.

Citation 
@misc{plumbline_0_6b_2026,
  title        = {Plumbline-0.6B: rule adherence from machine-verifiable supervision},
  author       = {Kav-Venaki, Eitam},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/dsup2026/plumbline-0.6b}},
  note         = {LoRA adapter over Qwen/Qwen3-Reranker-0.6B}
}

Results here are reported on LegalBench, which should be cited if you quote them: 

@misc{guha2023legalbench,
  title        = {LegalBench: A Collaboratively Built Benchmark for Measuring Legal
                  Reasoning in Large Language Models},
  author       = {Guha, Neel and Nyarko, Julian and Ho, Daniel E. and R{\'e}, Christopher
                  and Chilton, Adam and Narayana, Aditya and others},
  year         = {2023},
  eprint       = {2308.11462},
  archivePrefix= {arXiv},
  primaryClass = {cs.CL},
  url          = {https://arxiv.org/abs/2308.11462},
  note         = {40 authors; the seven contract\_nli tasks are used here}
}

If you cite this, please also cite the base model, which does most of the work: 

@article{qwen3embedding,
  title={Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models},
  author={Zhang, Yanzhao and Li, Mingxin and Long, Dingkun and Zhang, Xin and Lin, Huan and Yang, Baosong and Xie, Pengjun and Yang, An and Liu, Dayiheng and Lin, Junyang and Huang, Fei and Zhou, Jingren},
  journal={arXiv preprint arXiv:2506.05176},
  year={2025}
}

Runs of dsup2026 plumbline-0.6b on huggingface.co

216
Total runs
10
24-hour runs
49
3-day runs
84
7-day runs
216
30-day runs

More Information About plumbline-0.6b huggingface.co Model

More plumbline-0.6b license Visit here:

https://choosealicense.com/licenses/polyform-noncommercial-1.0.0

plumbline-0.6b huggingface.co

plumbline-0.6b huggingface.co is an AI model on huggingface.co that provides plumbline-0.6b's model effect (), which can be used instantly with this dsup2026 plumbline-0.6b model. huggingface.co supports a free trial of the plumbline-0.6b model, and also provides paid use of the plumbline-0.6b. Support call plumbline-0.6b model through api, including Node.js, Python, http.

plumbline-0.6b huggingface.co Url

https://huggingface.co/dsup2026/plumbline-0.6b

dsup2026 plumbline-0.6b online free

plumbline-0.6b huggingface.co is an online trial and call api platform, which integrates plumbline-0.6b's modeling effects, including api services, and provides a free online trial of plumbline-0.6b, you can try plumbline-0.6b online for free by clicking the link below.

dsup2026 plumbline-0.6b online free url in huggingface.co:

https://huggingface.co/dsup2026/plumbline-0.6b

plumbline-0.6b install

plumbline-0.6b is an open source model from GitHub that offers a free installation service, and any user can find plumbline-0.6b on GitHub to install. At the same time, huggingface.co provides the effect of plumbline-0.6b install, users can directly use plumbline-0.6b installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

plumbline-0.6b install url in huggingface.co:

https://huggingface.co/dsup2026/plumbline-0.6b

Url of plumbline-0.6b

plumbline-0.6b huggingface.co Url

Provider of plumbline-0.6b huggingface.co

dsup2026
ORGANIZATIONS
huggingface.co

Other API from dsup2026