G-reen / model-family-detector

model-family-detector (final-t001)

A classifier that reads a piece of long-form English writing (essay, article, review, story, blog post) and predicts which LLM family wrote it — OpenAI, Anthropic, Gemini or Other — with an auxiliary AI-vs-human output. It is a LoRA adapter on Gemma-3-4B-it with a small two-task head. It is the first complete model of this project ( final-t001 , trained 2026-10-10/11); it is useful, but the Results and Known issues sections below describe real weaknesses on the newest model generations, and you should read them before relying on it.

TL;DR of the numbers (top-1 accuracy on AI-written text, canonical normalisation): 93% on held-in sources, 63% on sources never seen in training (4 classes) , 54% on the LEDE benchmark 4-way / 74% on LEDE when forced to choose among the three real families , 89% on the IntelLabs peer-review benchmark. AI-vs-human AUROC 0.999 on every validation set — but the AI head calls ~50–60% of GPT-5-family long-form text "human" on LEDE. Details below.

Contents of this repo

How to use

pip install "transformers>=4.57" "peft>=0.17" torch huggingface_hub
# request access to this repo on the Hub first (manual gating), then:
python inference.py --repo G-reen/model-family-detector \
  --text "The quarterly numbers tell a simple story: revenue is up, margins are flat, and the board is nervous."
# p_ai=0.952 family=Other 3way=Gemini OpenAI=0.10 Anthropic=0.12 Gemini=0.14 Other=0.64 | 'The quarterly numbers ...'
# (a one-sentence input like this is far below the >=150-word texts the model was trained on; the family
#  distribution is close to flat and lands on Other. Give it a full article or essay.)
python inference.py --repo G-reen/model-family-detector --jsonl texts.jsonl --out preds.jsonl   # {"text": ...} per line

import inference                                  # from this repo
model, tok, max_len = inference.load("G-reen/model-family-detector")      # needs a GPU with ~10 GB; bf16
preds = inference.predict(model, tok, ["...text 1...", "...text 2..."], max_len)
# each pred: {"p_ai": <float>, "p_family": {"OpenAI": ., "Anthropic": ., "Gemini": ., "Other": .},
#             "family": <argmax of p_family>, "family_3way": <argmax over the three real families>}

The base model ( unsloth/gemma-3-4b-it , a re-upload of Google's Gemma 3 4B IT) is downloaded automatically; its vision tower is loaded but unused. Inputs are truncated to 768 tokens (the training length); the model reads the first ~550 words of a text. Texts shorter than ~150 words are outside the training distribution.

Reading the outputs

The head emits 5 logits from one linear layer , read as two independent quantities:

• p_ai = sigmoid(logit 4) — probability the text is AI-written. Trained on all rows (human rows included).
• p_family = softmax(logits 0..3) — P(family | the text is AI-written) . Trained only on AI rows ; human rows are masked out of this loss. It always sums to 1 and says nothing about human-vs-AI; on a human text it is meaningless.

So: read p_ai first; if it says AI, read the family. The joint probability of "AI and written by family k " is p_ai * p_family[k] .

family_3way is the argmax over OpenAI/Anthropic/Gemini only. The Other class absorbs most of the model's uncertainty on unfamiliar text (see Results); on the LEDE benchmark the forced 3-way reading is right 74% of the time where the 4-way reading is right 61%. Use it when you know the text came from one of the three big labs.

By default inference.py passes the text through regularizer.canonical() (curly quotes → straight, dashes → ASCII, markdown emphasis/headers/bullets stripped, rule lines removed, whitespace normalised). The model was selected on canonical text and its raw-vs-canonical scores are within a point of each other, so either is fine; --raw skips it.

Classes

Gemma note. This adapter was trained with Gemma mapped to Other (a decision taken 2026-10-04). Evaluation showed that is the wrong label: the model calls gemma-3-27b-it "Gemini" 65–67% of the time and gemma-2 66–72%, because Gemma writes with the Gemini fingerprint. The project switched to Gemma = Gemini on 2026-10-11; the published step-1/step-23 datasets and this adapter still reflect the old labelling. If your Gemma text comes back as Gemini, the model is right and the label was wrong.

Architecture

• Backbone : Gemma-3-4B-it (4.3 B params), frozen, bf16, SDPA attention. Sequence representation = hidden state of the last non-pad token (HF Gemma3ForSequenceClassification pooling).
• LoRA : r = 32, α = 32, dropout 0.1, rsLoRA, on q_proj k_proj v_proj o_proj gate_proj up_proj down_proj of every text layer (vision tower and projector excluded). 60.3 M trainable parameters incl. the head.
• Head : Linear(2560→256) → GELU → Dropout(0.1) → Linear(256→5, no bias) , fp32. No LayerNorm in this configuration.
• Loss : 0.3 · CE(logits[0:4], family) on AI rows only + 0.7 · BCE(logit[4], is_ai) on all rows. The 0.3 family weight is a project decision (the AI/human task was meant as regularisation, not the product).

Data

Everything is built from public datasets plus the project's own G-reen/fastdetector-train-stat-train (no API generation was done). The pipeline is documented in detail in the dataset cards; the short version:

1. Step 1 — unify ( G-reen/model-family-step1-train / -test ): 36 loaders, 3,472,346 rows, one schema ( text, model_family, model_id, metadata with 19 identical metadata keys). Gemma → Other; AI-edited text kept unless it is a near copy of its human source (cosine distance < 0.1 to the original removed, matching the fastdetector filter).
2. Step 2 — keep writing only : a rule pre-filter (length, structure) then a gemma-4-E4B-it judge that keeps only standalone long-form writing (essays, articles, stories, reviews, letters, blog posts); chat answers, code, math, lists, Q&A, dialogue scripts, forum threads and templated bot output were removed. For chat-derived sources a second judge also saw the user prompt and dropped rows where the user had not asked for a piece of writing.
3. Step 3 — clean : mojibake repair, chat-template tokens, HTML/markdown wrappers, acknowledgement openers ("Sure, here is…"), bylines, "THE END"/word-count footers, refusals, placeholders, self-references ("as an AI language model"), degenerate repetition, near-duplicates (MinHash), AI rows that copy their human original's structure, unsafe content. Titles, closing offers and typography were deliberately not removed — they are handled by the training-time regularizer so the model does not learn them as shortcuts. Two independent audits (reports in the project) drove versions 2 and 3. Result ( G-reen/model-family-step23-train , v3, 2026-10-08): 730,625 rows — Human 303k (aux head only), Other 200k, OpenAI 119k, Anthropic 76k, Gemini 32k, across 24 source configs.
4. Step 4 — splits and balancing ( training_code/build_step4.py ): three whole sources are held out as leave-sources-out validation ( val_loco : imbue_demo, lmarena_100k, ghostbuster_news — 12,500 rows, 2,500 per class incl. Human); a 5% grouped in-distribution validation split ( val_id , 11,255 rows; grouped by prompt/source key so paraphrase pairs never straddle the split); the rest is train: 650,541 rows (OpenAI 101,680 / Anthropic 64,694 / Gemini 23,164 / Other 186,952 / Human 274,051). Balancing is done by sampling weights, not by dropping rows: AI rows are weighted n_cell^-0.6 · N_family^-0.5 (cell = source × family), and 35% of every batch is Human text (for the AI head only).

Train rows by source (step-4 train split):

Held-out (never trained on): imbue_demo (claude-sonnet-4-6, gemini-3-flash-preview, gpt-5.3-chat-latest and human text in reviews/news/essays/stories; cc-by-nc-sa-4.0), lmarena_100k (40+ open and closed 2024 models), ghostbuster_news (gpt-3.5-turbo, Claude 2023, Reuters human text).

Test sets ( G-reen/model-family-step23-test ): LEDE (146,820 rows: 24 models of 2024–2026 in the generated and we = writing-enhancement slices; OpenAI 63.5k, Anthropic 42.2k, Gemini 34.6k, Other = gemma-3-27b-it 6.6k), IntelLabs peer-review benchmark (39,800 rows; GPT-4o, Claude, gemini-1.5-pro-002, llama-3.1-70b, qwen2.5-72b, human reviews; OOD genre, not uploadable), tell-human-detectors (300 rows: plain / paraphrased / humanized GPT-4o and Claude text, human). Test sets were evaluated once , on the finished model, with a cap of 7,500 rows per class.

Training process

• Regularizer ( regularizer.py ): at training time each example independently gets, with probability scaled by reg_strength = 0.5 , any of 20 transformations — quote/dash/ellipsis normalisation, markdown and header stripping, bullet and rule-line removal, title line removal/insertion, closing-offer removal, paragraph re-flow, double spaces, case changes, unicode hyphens — so that typography and boilerplate cannot be used as family shortcuts. Random chunking ( chunk_p ) was searched but the chosen configuration has it off.
• Stage-1 sweep ( mf-broad-1 , 84 Optuna trials, TPE + Hyperband, 12k rows each, objective = leave-sources-out canonical macro-F1): learning rate, batch, LoRA rank/α/dropout/rsLoRA, head norm/bottleneck, weight decay, warm-up, scheduler, max length {512, 768}, regularizer strength, chunking, balancing exponents, human share, base {pt, it}. Best 0.584.
• Stage-2 sweep ( mf-narrow-1 , 21 trials, 120k rows each, narrowed space): three trials completed at 0.58–0.59; the held-out score plateaued from ~40k rows while in-distribution kept rising (0.84 → 0.96) — more rows of the same sources teach source style, not family style. Trial t001 was chosen for the best score on the imbue_demo source (0.66) and the most balanced per-class recall.
• Final run ( final-t001 ): t001's configuration — lr 3.66e-5 constant with 3% warm-up, effective batch 16, weight decay 0.1, 768 tokens, bf16, gradient checkpointing — on 300,000 weighted samples drawn from the 650k train rows (≈ one pass), 4 × A6000 with torchrun DDP, ≈ 10 h wall (≈ 40 GPU-hours). Evaluation every 25k rows on 1,000 rows per class; the end-of-training checkpoint is the released one (no checkpoint selection: the held-out curve is flat within noise after 100k rows). Provenance note : the run was interrupted once by a cluster hand-over and resumed from its 125k-row checkpoint; a duplicate resume later replayed rows 250k–300k from the same checkpoint with the same data order. The released weights are one coherent 300k-row trajectory.

Results

All numbers are top-1 accuracy on AI-written rows unless stated; "canon" = text passed through canonical() , "raw" = as published. Validation rows: 1,500 per class.

LEDE confusion, canonical (rows = true class, 7,500 per class; Other = gemma-3-27b-it, 6,583):

            OpenAI  Anthropic  Gemini  Other
OpenAI        3170       443    1105   2782
Anthropic      179      6031     781    509
Gemini         341       313    5362   1484
Other(Gemma)   356       956    4250   1021

Per-model accuracy on a 24k-row random LEDE sample (canonical; →Other = share sent to Other; P(AI)<.5 = share the AI head calls human):

On the 22,500 OpenAI/Anthropic/Gemini LEDE rows: 4-way 0.607, forced 3-way 0.743 canonical / ≈0.80 raw ; when the model names a real family rather than Other it is right 80% of the time at 76% coverage. Raw text helps OpenAI most (gpt-5.4-nano forced 3-way 0.94 raw).

Sanity checks on shortcuts: a gradient-boosted classifier on 28 surface statistics (lengths, punctuation rates, markdown counts, type-token ratio) reaches 0.94 AUROC for AI-vs-human on the held-out sources but only 0.35 macro-F1 for the family task (vs 0.63 for this model), so the family signal is not punctuation and length. In-distribution scores are inflated by single-family sources (wildchat = OpenAI only, norquinal = Anthropic only, …); the held-out numbers are the honest ones.

Known issues (read before using)

1. The AI head misses GPT-5-family long-form text. On LEDE, 19.5% of AI texts get p_ai < 0.5 ; gpt-5 / 5-nano / 5.4 / 5.4-nano / 5.4-pro are called human 49–64% of the time, o3 34%, o1 21%; Claude and Gemini 1–4%. On LEDE's writing-enhancement slice, 69% of OpenAI rows are called human. The validation sets did not contain these models and reported AUROC 0.999. Do not gate long-form text on p_ai for OpenAI-2025 models.
2. The newest frontier generations are under-covered outside fiction. claude-opus-4-6 (0.42), claude-sonnet-4-6 (0.53), gemini-3.1-flash (0.19) and the GPT-5 family are the weak cells; in training these generations come almost entirely from storyscope (fiction). The model's reaction to "unfamiliar" text is Other (and, for OpenAI, human).
3. Other is learned as "Llama-like", not "none of the above." Held-out Llama variants are Other 65–72% of the time, but Reka, Yi, Athene, Command-R go to OpenAI 67–70% and DeepSeek 53%.
4. Gemma is labelled Other in this adapter but behaves as Gemini (see Classes).
5. Scores are for English, ≥150-word prose, truncated at 768 tokens; chat answers, code, lists and very short texts were filtered out of training and are untested.
6. Human text in the validation data (news, essays, Wikipedia-style, Common-Crawl prose) is much easier than modern edited human writing; expect a higher false-AI rate on polished human Substack-style text than the 0.2% measured here.

Data and licensing

The adapter is a derivative of Gemma 3 and is subject to the Gemma Terms of Use . The training data mixes licences from permissive (MIT, apache-2.0, cc-by, ODC-BY) to non-commercial (cc-by-nc-sa, cc-by-nc-nd, editlens access terms) and no-redistribution (LMSYS-Chat-1M); NYT-derived and Books3-derived prompts appear upstream of several sources. The weights are therefore released for research use only , behind manual gating, like the datasets. The IntelLabs peer-review benchmark is used for evaluation only and is not redistributed.

Reproducing

python -m training_code.build_step4            # step-23 parquet -> step-4 splits (set MD_STEP23V3_DIR / MD_STEP4_DIR)
python -m training_code.sweep --stage broad --study mf-broad-1 --trials 6     # per GPU worker; see sweep.sbatch
python -m training_code.sweep --stage narrow --study mf-narrow-1 --trials 2 --inherit narrow.json
torchrun --nproc_per_node=4 -m training_code.train $(cat launch args in launch_final.sh)
python -m training_code.predict --run <run dir> --split test_lede --limit 24000 --out preds.parquet

The code expects the fastdetector package on PYTHONPATH for the Optuna study helper only; everything else is in training_code/ .

Changelog

• 2026-10-11 — final-t001 released. Known: Gemma labelled Other; GPT-5-family AI-head miss. Next planned: retrain with Gemma = Gemini ( step4_gemma data), non-fiction coverage of 2025 models.