oddadmix / Jisr-Align-29M

Jisr-Align-29M

The encoder half of the Jisr Arabic-English MT model, contrastively finetuned for sentence alignment . 29M parameters, 448-dim, 6 layers. Trained from scratch.

Pair it with the DP search in align_dp.py (see align-documents.py ) to align Arabic and English documents into 1-1, 1-many, many-1 and null links with a calibrated confidence. On its own it is a bitext-mining encoder: mean-pool hidden_states[5] over the attention mask, subtract the per-language or per-document mean, L2-normalise.

Results

Held-out strict F1 (exact span match, swap-affected links excluded), against the untrained warm start and a length-only Gale-Church DP over the same search:

FLORES 1012-way ar->en retrieval P@1 is 1.0000 (warm start: 0.9763), which is why it is not the model-selection metric - it saturates by step 1000 of 19511. Selection ran on mean strict alignment F1 over held-out dev gold instead.

Gold sets: oddadmix/jisr-align-gold .

Two things it does not do

Reordering. The search is monotone. Sentences that swap order across the translation are unrecoverable by construction, and the headline metric excludes them rather than pretending otherwise.

2-2 links. Recall is ~0.125-0.250 and the finetune did not move it, because it was never an encoder problem: a (2,2) step costs 3.03x under nm_pow=0.8 while two (1,1)s cost ~2, so the DP can essentially never choose one. That needs the cost structure changed, not more training.

Training

One epoch of symmetric InfoNCE with in-batch negatives over oddadmix/quick-mt-en-ar-5m (10M directed pairs), batch 512, learnable temperature clamped CLIP-style, lr 3e-4 cosine, bf16, ~35 minutes on one GPU. No dialect data and no hard-negative mining - both are deliberately held back as separate measured ablations. The +0.196 gain on dialectal Arabic came from general representation quality, not dialect adaptation.

Usage

import numpy as np
import torch
from transformers import AutoTokenizer, MarianMTModel

M = "oddadmix/Jisr-Align-29M"
tok = AutoTokenizer.from_pretrained(M)
enc = MarianMTModel.from_pretrained(M).model.encoder.eval()

TAG = {"ar": ">>ara<<", "en": ">>eng<<"}   # tag each side with its OWN language

@torch.no_grad()
def embed(texts, lang, layer=5):
    # Masked mean-pool, centre, L2-normalise. Ready for cosine.
    b = tok([f"{TAG[lang]} {t}" for t in texts], return_tensors="pt",
            padding=True, truncation=True, max_length=256)
    h = enc(**b, output_hidden_states=True).hidden_states[layer]
    m = b["attention_mask"].unsqueeze(-1).to(h.dtype)
    x = ((h * m).sum(1) / m.sum(1)).float().numpy()
    x = x - x.mean(0, keepdims=True)          # per-language centring
    return x / np.maximum(np.linalg.norm(x, axis=1, keepdims=True), 1e-9)

ar = embed(["تطفو الإبرة الفولاذيّة على الماء بسبب التوتّر السطحي.",
            "القطة تجلس على السجادة.",
            "ذهب إلى السوق في الصباح الباكر."], "ar")
en = embed(["The cat sits on the mat.",
            "He went to the market early in the morning.",
            "The steel needle floats on water because of surface tension."], "en")

print((ar @ en.T).argmax(1))    # -> [2 0 1], the correct permutation

Three details in there are load-bearing, and each was measured:

Layer 5, not the last. The final layer is specialised for decoder cross-attention and scores worse at every tag setting. hidden_states[5] on a 6-layer encoder is the penultimate one.

Centring is the single biggest lever and it is free. Subtracting the per-language mean moves FLORES ar->en P@1 from 0.939 to 0.973, and from 0.241 to 0.970 at the worst tag setting. Two 448-dim vectors, no training.

Centring needs a batch. x.mean(0) over a single sentence is that sentence, so embed(["..."], "ar") returns a zero vector - a silent, total failure. Pass a whole document or corpus side at once, or fit the mean once on a sample and subtract that instead.

© KAND CA 2026