oddadmix / Nawah-Dialect-BERT-6M

Nawah-Dialect-BERT-6M — مصنّف اللهجات العربية بمُرمِّز BERT

A 5.98M-parameter Arabic dialect identifier. Give it a piece of Arabic text; it returns one of 14 labels — 13 dialects or Modern Standard Arabic.

Same task, same data, same evaluation as the Llama-based Nawah-Dialect-500K / Nawah-Dialect-v1 — this run swaps the backbone for Nawah-BERT-6M-v2 , a bidirectional BERT encoder, to see how a small encoder does on the same task against the existing causal-decoder ladder.

import torch
from transformers import AutoTokenizer
from dialect_model import DialectClassifier, predict     # ships in this repo

M = "oddadmix/Nawah-Dialect-BERT-6M"
tok = AutoTokenizer.from_pretrained(M)
model = DialectClassifier.from_pretrained(M)

predict(model, tok, "انا رايح السوق دلوقتي واجيب حاجات من هناك")[0]
# {'label': 'eg', 'name': 'Egyptian', 'score': 0.918}

Strip diacritics before inference — trained on undiacritized text, same as the rest of this family. Mean-pool, not [CLS] — Nawah-BERT-6M-v2 was pretrained on packed 2,048-token chunks with no meaningful CLS position, so this repo does not use AutoModelForSequenceClassification 's default pooler; dialect_model.py 's DialectClassifier wraps the bare backbone with mean pooling over non-pad positions and a small 2-layer scorer instead — the same trap documented on the backbone's own model card.

Labels

14 classes — 13 Arabic dialects plus Modern Standard Arabic:

ma Moroccan · eg Egyptian · dz Algerian · sa Saudi · msa MSA · sd Sudanese · bh Bahraini · tn Tunisian · lb Lebanese · ye Yemeni · sy Syrian · ps Palestinian · iq Iraqi · ly Libyan

Results

Test set: 3,079 held-out rows, identical split to the Llama-based models. Majority-class baseline 0.140 .

The 6M BERT lands within 0.13 points of accuracy and 0.53 of macro-F1 of the 51.8M Llama model — essentially matching it — at 1/8.6 the size , and clearly beats the 500K model on both metrics.

Also roughly half the size of the two dialect-router releases

Outside this Llama-based ladder, two earlier dialect-ID models were released as part of the Lahgtna TTS routing pipeline: dialect-router-v0.1 and dialect-router-v0.2 , both fine-tunes of asafaya/bert-mini-arabic at 11.55M parameters — essentially double this model's size:

Not a strictly apples-to-apples comparison — different training corpus, different test set, and dialect-router-v0.2 also carries an English label this model doesn't — but on their own reported numbers, this model matches or slightly beats dialect-router-v0.2 at about half the parameters . ( dialect-router-v0.1 reports no accuracy/F1 in its card, so it isn't in the table.)

Per-class accuracy (worst first)

The same short-text-starved classes that were weak on the 51.8M model ( ps , sy , ly , ye ) are weak here too — this is a data-scarcity property of the task, not something specific to this backbone.

Confusions are linguistic, not artefactual

Top confusions: ye→bh (8), ps→sy (7), sd→ye (7), sy→sd/ps/lb (6/6/5), bh→ye (6), iq→bh (5), sd→bh/ps (6/5) — all within one dialect family (Levantine: ps / sy / lb ; Gulf and African: bh / ye / sd / iq ). Nothing confuses across family boundaries.

The register confound, checked

The MSA class mixes transcript rows with chunks of written articles (see prepare_dialect.py ). If the model were reading written-vs-spoken register instead of MSA , the transcript slice would collapse relative to the article slice:

0.968 on transcripts — matching the 51.8M model's reported 0.968 on the same check almost exactly. The model is reading MSA, not register.

Limitations

Same as the rest of this family: inputs shorter than ~25 words are unreliable across all classes; ps / sy / ly / ye are under-resourced in the source corpus; source text is YouTube speech transcripts, so performance on formal writing, social media orthography, or Arabizi is untested.

Training

Base Nawah-BERT-6M-v2 — BERT, hidden 128, 8 layers, 2 heads. Data prep is prepare_dialect.py , unchanged from the Llama-based models (it is backbone-agnostic — text normalisation, dedup, and the train/test split don't depend on what reads the text afterward): 573,829 train / 3,079 test rows from oddadmix/dialectal-arabic-text-no-audio-v2 (YouTube transcripts) plus MSA chunks from oddadmix/msa_texts_summaries_new .

train_dialect_cls_bilingual.py 's MeanPoolClassifier : bare AutoModel backbone → mean pool over non-pad positions → Linear(h,h) → GELU → Linear(h, 14) , cross-entropy loss. LR 3e-4 cosine, batch 64, 3 epochs, bf16, max_length 192, checkpoint selected on macro-F1. 26,901 steps, 6m16s on a single consumer GPU.

prepare_dialect.py and train_dialect_cls_bilingual.py are in this repo.

© KAND CA 2026 — PROJECT NAWAH