enver / rootformer-v20.2-mt

Rootformer v20.2 — Arabic→English: leak-free evaluation and the root-awareness verdict

v20.2 is the evaluation release. It contains the honest Arabic→English test harness, the leak-free split, the baselines, and a controlled test of whether the Farāhīdian root representation helps translation. It does not — in either form tested — and this release documents that with the evidence.

Companion: the work-level leak-free split is published as a dataset, enver/ar-en-rootformer-leakfree-split .

The result

Evaluated on held-out works (31 whole works absent from training; 17,916 test sentences whose 13-grams never occur in train). Metrics: chrF++ / BLEU (sacrebleu).

Reading it.

1. The shipped rule engine scores below the identity floor. It generalises not at all; its published benchmark was memorisation.
2. In-domain data is the real driver: +16.7 chrF++ over zero-shot.
3. Textual root conditioning: −0.28 chrF++ — no gain.
4. Architectural root conditioning: no gain, and provably inert. At 6,000 steps the model scores identically (45.91 chrF++) whether it receives the sentence's own root vectors or another sentence's roots , with the shuffled arm even edging BLEU. It cannot distinguish correct roots from mismatched ones, so the conditioning contributes nothing. The +0.12 over the matched no-conditioning control comes from the extra parameters and steps.

Two independent mechanisms, both negative, on a split built so that memorisation cannot help.

What is included

Method notes

• The split holds out whole works , never sentences from a work that also contributes training pairs. Raw overlap before filtering was 0.98 % (Arabic) / 1.33 % (English); the published test_clean removes every offending row, leaving 0.000 % .
• Both conditioning forms are compared against a matched no-conditioning control and a shuffled/null control , from the same checkpoint, on the same data slice, with the same schedule.
• COMET is not reported. Installing unbabel-comet pulled numpy 1.26.4 over the working numpy 2.1.2 and broke the environment; it was reverted and repaired rather than risk the host. The metrics are chrF++ and BLEU only.
• The English references are machine translations ( *_v4_translated ), which caps absolute scores. Comparisons between systems on identical references remain valid; a human-translated test set is needed for a publishable "high-grade" claim.

Correction notices carried forward

• The v19.2 "transmutation" was a hardcoded dictionary plus sentence regexes, not model output — it returns byte-identical text under real hidden states, zero hidden states, None , and hidden states from a different sentence.
• The Grand-100 LaBSE score of 0.9037 is memorisation — all 100/100 Arabic sentences and 100/100 English references occur verbatim in the training corpora.
• One bilingual corpus is randomly paired — an independent-MT agreement audit with a shuffled null gives pure_gold +17, sovereign_classical_transmute +21, unified_basran_andalusian +20, but grand_scholastic_bilingual +4.7 (effectively random).

Citation

@software{rootformer_v20_2_2026,
  author = {Enver at Aynengine and the Farāhīdian Research Circle},
  title  = {Rootformer v20.2: Arabic-to-English Leak-Free Evaluation and the Root-Awareness Verdict},
  year   = {2026},
  url    = {https://huggingface.co/enver/rootformer-v20.2-mt}
}