desert-ant-labs / toxic

toxic: on-device multilingual hate-speech and abuse triage

A multilingual toxicity classifier for on-device hate speech detection and content moderation triage: five content heads ( HATEFUL , HARASSMENT , THREAT , SEXUAL , SELF_HARM ) plus protected-group target heads, measured on eight languages (en, es, pt, fr, de, it, nl, pl). Four of them (English, Spanish, Portuguese, French) rank among the world's ten most spoken languages; together the eight are spoken by well over a billion people worldwide. The deployable model is ~66.7 MB (4-bit Core ML on Apple, 2.8 ms on an M3 Ultra Neural Engine), ~75.5 MB (int4 LiteRT .tflite on Android and Linux), or ~83.3 MB (int4 ONNX toxic.onnx in the browser, bit-exact under ONNX Runtime Web). Inference is 100% on-device.

At its size, this is the strongest measured system we know of for hate speech detection in these eight languages, and it is competitive with models an order of magnitude larger. Every runnable competitor was re-run through the identical benchmark harness. On the seven non-English languages, toxic beats every one of them, including the full-precision server builds of guard models 6 to 14 times its size, by 13 to 30 points of average macro-F1. On English, the 134.9 MB int8 build outscores every competitor's shipping 4-bit quant (0.808 vs 0.794 for Llama-Guard-3-1B int4 at 955 MB); the 75.5 MB int4 build shipped here gives up 0.007 on English to that 12.7x-larger model and nothing anywhere else. Full tables, including the cases competitors win, are in How it compares .

Throughout this card, quality numbers are macro-F1 on HateCheck and its multilingual extension: a public suite of roughly 3,700 hand-written test cases per language, never trained on, built to trap classifiers with hard hate (slurs, spelling evasion, implied hate) and deliberately confusing non-hate (quotes, counter-speech, negation). Macro-F1 averages the score for catching hate with the score for letting clean text through, so a model cannot look good by simply flagging everything (or nothing); 1.000 is perfect.

Triage, not verdict. Outputs are escalation signals for human review or a heavier local tier, not autonomous removal decisions. SELF_HARM and THREAT are high-precision heads intended for resource routing with a human in the loop. The failure modes below are published on purpose: read them before wiring the model into any enforcement path.

Files

Taxonomy

Two multi-label heads over one shared encoder. Text can fire several labels at once; thresholds apply per label.

• Content heads (5): HATEFUL (scoped to the EU notion of illegal hate speech: public incitement to violence or hatred against a protected group, Framework Decision 2008/913/JHA), HARASSMENT (targeted insult / bullying of an individual), THREAT (threat of violence), SEXUAL (sexual harassment / unsolicited sexual aggression), SELF_HARM (self-harm / suicide content, for resource routing).
• Target heads (10): the 2008/913/JHA protected grounds RACE , COLOUR , RELIGION , DESCENT , NATIONAL_ETHNIC_ORIGIN , plus the extended grounds SEXUAL_ORIENTATION , GENDER , DISABILITY , AGE , OTHER . Only meaningful when HATEFUL fires; used for per-ground fairness reporting.

Architecture

• Encoder: XLM-R-base lineage (12 layers, hidden 768), vocabulary trimmed from 250k to 60,728 pieces (~133M params), fine-tuned with focal BCE on the two heads.
• Static shapes: fixed [1, 128] input, no dynamic control flow; ANE-native reformulation for the Core ML export, dynamic-range quantization for LiteRT.
• Runtime: tokenize off-graph (SentencePiece-Unigram, tokenizer.json or toxic_tokenizer.bin ), run the encoder, apply sigmoid per label, threshold per the menu below. Default threshold 0.40 .

Measured quality (the shipped artifacts, not the checkpoint)

Every number below is the full Multilingual HateCheck scorer re-run on the exact artifact in this repo, threshold 0.40. HateCheck is eval-only: it is never trained on. Torch reference: 8-language mean macro-F1 0.789 .

Phone-class latency (iPhone 13/15, mid-range Android) is not yet measured; the M3 Ultra figure bounds the graph cost.

The web row is the one platform where the deployed runtime was also verified numerically: under ONNX Runtime Web 1.27 (wasm EP) the browser reproduces the native toxic.onnx scores bit-for-float (max probability delta 1.1e-6 over a 120-fixture, 8-language drift study, zero decision flips), so the browser ships exactly the 0.775 measured here (0.774 on the seven non-English languages). toxic.tflite stays Android-native: its 0.786 was measured with native LiteRT kernels, which browser wasm kernels do not reproduce. The web SDKs pin release tag v0.1.1 (the first tag that carries toxic.onnx ).

Per language (threshold 0.40):

How it compares (measured, same scorer, same suite)

Every runnable competitor was re-run through the identical HateCheck harness, in both its full-precision (bf16 / fp32) build and its measured int4 quant (bitsandbytes nf4), so quantized is compared against quantized and full against full. Generative guards are scored on their parsed safety verdict (Unsafe = hate); their on-disk sizes are published Q4_K_M GGUF bytes. 7-EU = mean over the seven non-English languages (de, fr, nl, pt, es, it, pl). Hate recall = share of hateful cases caught; non-hate FPR = share of non-hateful cases wrongly flagged.

Per language, macro-F1, each system in its listed build:

What the table says:

• Seven of eight languages: toxic wins outright at 6 to 14x less disk , against full-precision and int4 competitor builds alike. The margin is never small: +10.4pp over the best competitor build on de, +12.5pp fr, +10.1pp nl, +11.6pp pt, +10.7pp es, +13.4pp it, +18.4pp pl (shipped .tflite vs each competitor's best build in any precision).
• English: Llama-Guard-3-1B leads in bf16 (0.814 vs toxic's 0.810 reference). At shipping quants the picture flips with size: toxic int8 (134.9 MB) 0.808 beats Llama-Guard int4 (955.4 MB) 0.794; the shipped 75.5 MB int4 lands 0.787, 0.007 behind at 7.9% of the bytes.
• Quality per megabyte is not close: 1.04 to 1.16 macro-F1 per 100 MB for the shipped toxic artifacts vs 0.07 (Llama-Guard int4) and 0.13 (Qwen3Guard int4). That gap is what makes always-on, per-message, on-device moderation feasible at all.
• Detoxify multilingual, the incumbent same-architecture-class classifier, collapses outside its strong languages (de 0.299, nl 0.293, pl 0.243) and trails toxic by 15 to 20 points even where it works.

False positives in context

Two very different numbers describe "how often it flags clean text", and conflating them misreads the model.

On adversarial trap cases: 27 to 36%. HateCheck's non-hate cases are not ordinary text; they are traps built from the exact patterns that break hate classifiers. counter_quote / counter_ref quote or denounce hate in order to condemn it, slur_reclaimed is in-group reclaimed slur use, negate_neg is negated hate ("no one should ever hurt them"). At the default 0.40 threshold, toxic wrongly flags 27.1% (pl) to 35.9% (de) of these traps, depending on language.

On natural text: 2 to 3%. On the held-out test split of real corpora (Civil Comments, Measuring Hate Speech, GAHD; 14,021 clean rows never seen in training), the HATEFUL head fires on 3.0% of clean rows overall. On 13,559 clean English user comments (Civil Comments) it is 2.3% for HATEFUL and 7.3% for any of the five heads (the extra fires are almost all HARASSMENT at 6.4%; THREAT 0.4%, SEXUAL 0.4%, SELF_HARM 0.0%). On clean Measuring Hate Speech rows: 3.4%. The one natural-corpus outlier is German GAHD at 38.8%, and that is by design: GAHD is an adversarially constructed dataset of contrast pairs (quotes, counter-speech, borderline rephrasings), so it measures the trap rate, not the organic rate. Full breakdown: reports/RELEASE_extended.md in the training repo.

Beware the low-FPR trap when comparing. Llama-Guard-3-1B posts the lowest trap FPR (0.187 on the seven non-English languages, bf16), but it does so by barely firing: it catches only 59.4% of actual non-English hate in those languages, missing two in five hateful messages. toxic catches 87 to 88% at its operating points. A low false-positive rate bought by under-firing is not precision, which is why the headline metric is macro-F1 (balanced by construction) and why FPR numbers should always be read next to recall. If your application needs a lower FPR, buy it explicitly with the threshold menu below and keep the recall you are giving up on the table.

Threshold menu (per-language operating points)

Measured on the torch reference (the vocabulary trim is quality-neutral). macro-F1 is the HateCheck 2-class macro; FPR is the false-positive rate on non-hate cases, the number to watch if you fear over-blocking. Full sweep tables (0.30 to 0.70) live in the training repo ( reports/RELEASE_sweep.md ).

Read: raising the threshold trades hate recall for a lower non-hate FPR. At per-language tuned thresholds the seven-language multilingual average (all languages except English) is 0.794 and four languages (de, fr, pt, es) clear 0.80. The same menu ships in toxic_meta.json .

Failure modes (read before deploying)

This section is the practical read: what actually goes wrong, what it looks like in an app, and what to do about it. The metric tables backing every claim are in the deep-dive section below. Publishing these is part of the product.

1. It sometimes flags people quoting or condemning hate. News reports, moderation discussions, and counter-speech repeat the hateful words they are arguing against, and the model reacts to the words. A comment like "Someone in this thread said we should 'get rid of' immigrants, and that is disgusting" can get flagged as hateful even though it is the opposite. On trap suites this happens on roughly 1 to 5 cases in 10 depending on language, worst in English and German. If your app touches journalism, activism, or lets users discuss moderation, expect this. Mitigation: route flags to human review or a heavier model; never auto-remove on this signal alone.

2. Positive or neutral mentions of identity groups can trip it, mostly outside English. Sentences like "Proud to be a trans woman and having a great day" or "Many Muslim families live in our neighborhood" are sometimes flagged simply because an identity term appears. This is the dominant false-positive source in Polish, Dutch, German and Italian, where up to six in ten of such trap sentences fire. In natural text it is far rarer (the 2 to 3% above), but profile bios and community-pride posts concentrate exactly these sentences. Mitigation: use the higher per-language thresholds from the menu; treat short identity-statement texts as low-confidence.

3. Reclaimed slurs and lookalike words. In-group reclaimed usage (a community member using the term about themselves) is flagged in roughly 4 of 10 English trap cases, and harmless words that merely resemble slurs are occasionally hit too. Mitigation: human review for user-report flows in communities where reclaimed language is common.

4. It misses some slur-based hate in Dutch, German and Polish. Hate that consists of a bare slur with no other abusive phrasing slips through more often in these languages (only 4 to 6 trap cases in 10 caught, vs 7 in 10 for English). Mitigation: if slur coverage is critical, pair the model with a per-language slur wordlist; the two failure profiles are complementary.

5. Hate phrased as a question or an implication is harder. "Why do [group] always ruin everything?" and implied put-downs are caught less reliably than direct insults (roughly 6 to 9 in 10 depending on language, vs near-perfect on explicit threats). Mitigation: none needed for triage use; do not promise users that subtle hate is always caught.

6. Quantization costs, measured on the shipped artifacts.

• Core ML (iOS): the 4-bit compression costs about 3 to 4 points on English. fp16 on the ANE is quality-neutral (en 0.811 vs torch 0.810); the shipped 4-bit palettized model lands en 0.771 and 8-language 0.776 (-1.3pp mean vs torch). English carries most of the palettization loss.
• TFLite (Android): English pays -2.1pp for int4. toxic.tflite (int4 blockwise-32) scores en 0.787 vs 0.808 for the int8 build of the same model; the seven-language multilingual average is identical for both (0.786). If English dominates your traffic and 59 MB matters less than 2pp, ask for the int8 build.
• ONNX (web): the browser buys fidelity with -0.011 8-language F1 and +7.9 MB vs the TFLite artifact. toxic.onnx (int4e) measures 0.775 8-language natively, and ONNX Runtime Web reproduces those scores bit-for-float in the browser. The alternative was shipping toxic.tflite numbers the browser runtime never reproduces (wasm kernel drift up to 0.173 with 4.2% decision flips in the drift study); the web surface ships a number that is actually measured in the deployed runtime.

7. Not yet measured: latency on phone-class chips (iPhone 13/15, mid-range Android); the M3 Ultra and desktop-CPU numbers above bound the graph cost, not the thermal envelope.

The pattern behind 1 to 3: counter-speech, negated hate, reclaimed slurs and positive / neutral identity mentions are the dominant false-positive sources in every language; bare slurs are the dominant miss in nl, de and pl. If your application is FPR-sensitive, use the higher per-language thresholds above and route positives to review instead of auto-actioning.

Per-language deep-dive (per-functionality cells)

HateCheck groups its test cases into named "functionalities" (cells), each isolating one linguistic pattern. Numbers are accuracy on that cell for the flagship int8 artifact at threshold 0.40. Cells ending _h are hateful (an error is a miss); cells ending _nh are non-hateful (an error is a false positive). Decoder for the cell names used below:

Strongest and weakest cells per language:

Read: explicit threats, normalized violence, profanity-laden abuse and spelling evasion are near-perfect in every language; the errors concentrate in exactly the trap families of failure modes 1 to 5. Full per-cell tables for all eight languages are published in the training repo.

Fairness across protected groups

Two measured views, both on natural (non-synthetic) held-out data at threshold 0.40.

Does it recognize hate equally, whoever the target is? On the held-out test rows with human-annotated target groups (Measuring Hate Speech corpus), HATEFUL recall per protected ground:

Spread is 6.5 points (0.898 to 0.963): no ground is catastrophically worse, and none falls below ~0.90.

Does it over-fire on the mere mention of an identity? Joining the same test rows back to their raw identity-mention annotations gives, per group, two ranking scores (AUC: 1.0 = perfect separation, 0.5 = chance). "Mention trap" (BPSN) checks whether non-hateful text mentioning the group outscores other people's actual hate: low values mean the model over-fires on the mention. "Protection" (BNSP) checks whether hate against the group outscores other people's harmless text: low values mean the group is under-protected. Note the non-hate rows in this corpus are mostly still insulting or profane (it over-samples abusive text), so the FPR column is a stress number under matched toxicity, not the 2 to 3% natural rate above; the fairness signal is the spread across grounds against the same 0.448 background.

All grounds sit in a narrow band (BPSN 0.80 to 0.89, BNSP 0.81 to 0.90). The two mildest asymmetries, published rather than hidden: GENDER draws the most mention-driven over-firing (FPR 0.533 vs 0.448 background), and RACE has the lowest hate recall (0.898). Both are monitored targets for the next data iteration.

How these numbers were made

The evaluation discipline is the product as much as the weights are:

• Every quality number is re-measured on the exact shipped bytes , not extrapolated from the training checkpoint. Quantized artifacts are re-run through the full scorer per platform (Core ML, LiteRT, ONNX, torch).
• Benchmarks are eval-only. HateCheck and its multilingual extension were never trained on, in any language. The natural-corpus test split is carved by content hash before training and never touched.
• Experiments were pre-registered. Each of the ~30 logged training experiments stated its hypothesis, its single changed axis, and numeric falsifiers (the conditions under which the idea would be declared wrong) before the run started. Several were falsified and reverted; the record keeps both outcomes.
• Results were adversarially reviewed. Twenty methodology review passes (data leakage, confounds, over-generalization, severity audits of the synthetic data) ran against the experiment log during development, and their findings drove the next iteration.
• Ship candidates were replicated across three seeds ; the release model was selected on the seed band, not a lucky single run.
• Competitors run through the identical harness , same scorer, same threshold discipline, with their quantized and full-precision builds both measured rather than assumed.

License

Desert Ant Labs Source-Available License . Free for most apps; a commercial license is required at scale. Full terms at the link. Licensing: [email protected] .

Built exclusively from commercially clean components (CC0 / CC-BY / MIT / Apache-2.0 training data, MIT base encoder). Attributions, dataset citations (GAHD, Measuring Hate Speech, GermEval 2018, MLMA and others) and generator credits are in THIRD_PARTY_NOTICES.md .