enver / rootformer-v20.1-nrmt

Rootformer v20.1 — NRMT: an architecture fitted for Next-Root-Morph-Token prediction

v20.1 is the architecture release. It changes the model so that predicting the next root is a first-class autoregressive act instead of a byproduct of a hidden state, and it measures the change against a matched control.

Supersedes v20 (weights-only retrain). Weights: the NRMT heads here ( nrmt_head_full.pt ) sit on the v20 backbone checkpoint, which is unchanged and remains in enver/rootformer-v20 .

The problem this release fixes

The shipped head computed root_logits = W · h_t — a single linear map from the final hidden state. But next-root prediction is mostly a function of the recent root stream : a count-based 4-gram over roots reaches ≈ 33 % acc@1 where the head reached ≈ 7.5 %. The root history was never an explicit input. Six defects were identified and fixed:

Two further bugs were caught while validating: assigning the flash wrapper with self.flash = … silently registered it as a submodule , duplicating every flash parameter (581 phantom keys) and handing them gradients — fixed, now exactly 4 new tensors; and an unnormalised additive feature term drove held-out PPL to 3.0 × 10⁴ , fixed with LayerNorm + dropout.

The ablation — does the architecture help?

Identical data (12,000 windows = 1.52 M positions), identical protocol and hyper-parameters. use_features=False disables history + operator + morph conditioning and reproduces the plain W · h_t head.

Metric: next-root acc@1 on the file-level held-out stream, on ALL positions and on context-novel positions (12-root context never seen in train: 77.9 % of val positions).

The architecture wins at every matched step (+0.5 to +0.8 pp), on both ALL and novel contexts, and was still improving when the control had plateaued. Root train loss at 20 k steps: 4.13 vs 5.12.

Honest caveats

1. The gain is real but modest , not transformative.
2. PPL is worse for the full model (1296 vs 808) and rises with the feature norm (0 → 39.5). It is over-confident and needs label smoothing / temperature before its probabilities are trusted. Rank metrics (acc@1/@5) are the reliable signal here.
3. A count-based interpolated n-gram still beats both on novel contexts: 12.1 % vs 7.25 %. The learned head still extracts less transition structure than a lookup table. The next step is to feed the n-gram in as features rather than compete with it.
4. Not implemented: Sībawayh's valency/constituent stack ( Inqiṭāʿ al-ʿAmal ), al-Iktifāʾ clitic saturation, full jussive apocope in surface realisation, Marātib al-Maʿārif at decode time.

Correction notice — please read before citing earlier numbers

The v19.2 "transmutation" was a hardcoded dictionary + sentence regexes, not model output. Verified: the engine returns byte-identical text under real hidden states, zero hidden states, None , and hidden states from a different sentence.

The Grand-100 LaBSE score of 0.9037 is memorisation, not translation. All 100/100 Arabic sentences and 100/100 English references occur verbatim in the training corpora; the "unseen" half is not unseen. See BENCHMARK_CONTAMINATION.md .

One bilingual corpus is randomly paired. An independent-MT agreement audit with a shuffled-pair null gives: pure_gold +17, sovereign_classical_transmute +21, unified_basran_andalusian +20 (genuinely aligned) but grand_scholastic_bilingual +4.7 — effectively random pairs . The mujtahid_* and grand_scholastic_train sets are Arabic-only (100 % empty English).

Classical algorithms — correctness audit

Ten of twenty implementation checks failed against the primary texts and are now fixed. All primary sources are bundled under corpus/ .

Files

Usage

from nrmt_arch import RootformerNRMT            # + the v20 backbone
# load checkpoints/rootformer_v20_nrmp_master.safetensors from enver/rootformer-v20,
# then: model.nrmt_head.load_state_dict(torch.load("nrmt_head_full.pt"))

python nrmt_train.py --checkpoint <v20 backbone> --train-windows 12000 --steps 20000   # full
python nrmt_train.py ... --no-features                                                 # control
python test_grammar_impl.py     # 10/20 PASS on the original implementation
python verify_v2.py             # ALL PASS on the corrected implementation

Citation

@software{rootformer_v20_1_2026,
  author = {Enver at Aynengine and the Farāhīdian Research Circle},
  title  = {Rootformer v20.1: An Architecture Fitted for Next-Root-Morph-Token Prediction},
  year   = {2026},
  url    = {https://huggingface.co/enver/rootformer-v20.1-nrmt}
}