Muno459 / zipformer_p-arabic-v2

zipformer_p-arabic-v2

A streaming phoneme recognizer for Arabic. It maps audio to phonemes (consonant + haraka units) reflecting what was actually pronounced , including dialect and tajweed.

This is a broad Arabic model, not a Qur'an-only model.

It handles everyday Arabic (MSA, broadcast speech, and dialect: Egyptian, Gulf, Levantine, Maghrebi) and Qur'anic recitation with tajweed. Most Qur'an models collapse the moment you speak ordinary Arabic to them. This one does not. It is good at both.

It replaces two earlier models with one, and beats each of them on its own home turf, including the dedicated Qur'an model at Qur'an:

Read the middle column carefully. The Qur'an specialist is 39% worse on real-world Qur'an than this model, and it falls apart on ordinary Arabic (22.4% vs 4.7% CER) because it injects tajweed markers into everyday words. Combining the two domains cost nothing on Qur'an and gained everything elsewhere.

Specification

Why it handles noise better

v1 was trained on clean, un-augmented audio. v2 adds three augmentations, applied on GPU every batch:

This is not a claim, it is the clearest number in the table above. On real phone recordings of Qur'an (793 held-out clips, never seen in training) v2 scores 8.14% PER against 13.39% for the Qur'an specialist, a 39% relative error reduction on exactly the audio that clean-trained models struggle with. On clean studio recitation the gap is smaller (3.31% vs 5.82%), which is what you would expect: augmentation pays off most where conditions are worst.

One honest caveat: MUSAN is narrowband (median 99% energy bandwidth about 3.2 kHz, under 0.6% of energy above 6 kHz). The model is well hardened against low and mid frequency interference, which is what most real noise is, but it has seen comparatively little high frequency hiss.

Benchmarks

Official Qur'an-Lab benchmark (600 clips)

Scored with the official score.py normalisation. Phonemes are converted to text by nearest-ayah retrieval, the same method used for the published zipformer_p-quran entry.

Phoneme error rate, the metric retrieval does not mask

Read the PER, not the leaderboard WER. Nearest-ayah retrieval snaps output to one of about 9,112 ayat, so it is effectively a 9,000-way classification that already corrects most acoustic errors: retrieval picks the correct ayah 99.8% of the time for this model against 99.6% for the specialist. With only 0.4% of headroom, the leaderboard WER cannot show a large gain no matter how much the acoustics improve. A 28% relative PER improvement compresses into a 3% WER improvement. PER is the honest measure of the acoustic model.

General Arabic (non-Qur'an)

Reverberant YouTube khutbah (MSA sermon), phonemes converted to text by an LLM, scored against ElevenLabs Scribe:

Training data, about 2,039 h of unique audio

Broad Arabic, 1,527 h (MSA, broadcast, and heavy dialect coverage)

Qur'an, 512 h

Qur'an is upweighted 4x during training, so the model sees about 2,342 h per epoch (roughly 34% Qur'an).

In-house labels and datasets

The audio above comes from public and third-party corpora. What is ours is the labelling . There is no public Arabic phoneme corpus at this scale, so every phoneme label the model trains on was produced here, and released as two datasets:

• quran-phonemes , deterministic. Generated rule-based with quran-transcript (the same generator behind the muaalem gold), so the tajweed is exact rather than guessed: madd lengths, ghunna, idgham and qalqala all follow Hafs rules.
• arabic-broad-phonemes , LLM-phonemized and source-aware. Dialectal sources are prompted for dialectal vowels and MSA sources for MSA, deliberately preserving dialect instead of normalising it to MSA. Being LLM-generated it carries some noise, mainly vowel-length artifacts.
• Quality tiering. The tlog real-phone clips are tiered S, A and B by teacher-label confidence and weighted accordingly, so noisier labels count for less.

The tlog_holdout evaluation set is excluded from training, so the real-phone numbers above are measured on unseen clips, though they are the same domain, which is the fair way to read them.

The 41-symbol Qur'an alphabet is a superset of the 37-symbol broad alphabet, so both live in one 250-unit tokenizer.

Files

ONNX accuracy, measured not assumed

The ONNX graphs are true chunk-by-chunk streaming with cache states, so they are not bit-identical to full-utterance PyTorch decoding. That is by design. What matters is accuracy, measured on held-out real-phone Qur'an:

The 1.7 point gap is the normal streaming tax (limited lookahead against full-utterance context), not an export defect. INT8 costs only 0.23 points over fp32 while being 3.6x smaller, so int8 is the sensible default on device.

CoreML: cache-aware streaming, fp16 for the Neural Engine

Both CoreML builds are cache-aware streaming , produced from the same icefall wrapper as the streaming ONNX, so CoreML and ONNX are the identical per-chunk computation rather than two different models. Nothing is padded to a fixed utterance length.

Use the .streaming.ane.mlpackage (fp16) , since the Apple Neural Engine runs fp16 only. The fp32 package is a reference and CPU/GPU fallback.

Interface, per call:

fp16 is safe for this architecture, and that is measured rather than assumed. Two checks:

fp16 is safe for this architecture, and that is measured rather than assumed. Two checks:

This matters because a sibling model in this project (a FastConformer) does blank on ANE in fp16: its RelPositionalEncoding multiplies activations by sqrt(512), lifting peaks near 5,200 to about 118,000 and overflowing fp16. Zipformer2 does not have that multiply. It uses CompactRelPositionalEncoding , and the measured headroom above is roughly 63x, so the failure mode does not carry over.

One limit stated plainly: coremltools cannot execute a model outside macOS, and this was built on Windows and WSL, so the CoreML files are conversion-verified and precision-verified, but not runtime-verified on actual Apple hardware . The ONNX files were measured end to end against gold labels (table above). If you deploy on device, sanity-check the first outputs against the ONNX or PyTorch reference, and please open a discussion with what you find.

Per-phoneme confidence, alternatives and timings

Greedy CTC keeps the argmax and throws away the rest of the 251-way posterior. For recitation grading that discarded information is the useful part: "the model was unsure" and "the reciter said something different" look identical in a plain transcript. decode_with_confidence.py keeps it, so every emitted phoneme carries a confidence, the runner-up candidates, and a time span.

$ python decode_with_confidence.py clip.wav --topk 2

ءِننننَرَببَهُ۾۾۾بِهِميَومَءِذِللَخَبِۦۦر

         time  ph       conf  margin  alternatives
  1.20- 1.24  ننننَ   1.000   1.000  <blank>:0.00  نَ:0.00
  4.52- 4.60  ءِ      0.790   0.581  <blank>:0.21  ا:0.00
  5.44- 5.48  خَ      1.000   0.999  حَ:0.00  <blank>:0.00
  5.72- 5.80  بِ      0.894   0.789  <blank>:0.10  رِ:0.00

Each entry gives phoneme , confidence (0 to 1), margin (confidence minus runner-up), start / end in seconds, and alternatives . Pass --json out.json for machine-readable output.

Read the confidences with care. CTC posteriors are famously peaky: on the clip above the mean confidence is 0.983 and nothing falls below 0.5. So a naive confidence < 0.5 gate will almost never fire, and treating high confidence as proof of correctness will mislead you. What is actually informative:

• margin , not confidence . A phoneme at 0.79 with <blank> at 0.21 is a genuinely contested position; one at 1.000 with everything else at 0.00 is not.
• Which unit is the runner-up. <blank> competing means the model is unsure anything was said there (onset, pause, clipped sound). A real phoneme competing, for example خَ against حَ, means it heard a sound but is torn between two readings, which is the interesting case for pronunciation feedback.
• Thresholds must be tuned on your own audio , and are not calibrated probabilities. If you need true calibration, temperature-scale the logits against a labelled set of your own recordings.

Timing resolution is one encoder frame, 40 ms. That is enough to separate phonemes but coarse for grading a precise 2-harakah hold; treat the spans as boundaries, not fine duration measurements.

Usage

icefall Zipformer2 , 80-bin kaldi fbank (povey window) at 16 kHz, CTC greedy decode to phoneme units.

The feature extractor matters. Use torchaudio.compliance.kaldi.fbank , not torchaudio.transforms.MelSpectrogram . Feeding slaney-mel features roughly doubles the error rate (measured: 11.4% to 22.1% WER on the same audio). This is the most common way to get bad results.

import torch, torchaudio.compliance.kaldi as kaldi

feats = kaldi.fbank(wav, num_mel_bins=80, frame_length=25.0, frame_shift=10.0,
                    sample_frequency=16000, dither=0.0, snip_edges=False,
                    window_type="povey", use_energy=False, low_freq=20.0,
                    high_freq=-400.0, remove_dc_offset=True, preemphasis_coefficient=0.97)

For the 1000 ms look-ahead profile use chunk_size=(24,) and left_context_frames=(256,) .

The ONNX files are the standard sherpa-onnx online zipformer2-CTC format ( model_type=zipformer2 ), so they load directly in sherpa-onnx and in plain onnxruntime by feeding one chunk plus the cache tensors and carrying the returned states forward.

Limitations

• Phonemes, not text. Converting Arabic phonemes to orthography is a separate, hard problem. Judge this model by PER, not word-level text WER.
• Retrieval is a Qur'an-lexicon constraint. The leaderboard WER above uses nearest-ayah matching. It does not transfer to general Arabic.
• Dialectal text is not a strength. As a phonetic transcriber it keeps dialect faithfully, but reconstructing fluent dialectal orthography from it is out of scope.
• Broad-Arabic labels are LLM-generated and carry some vowel-length noise.
• The CoreML builds have not been run on actual Apple hardware (see above), since coremltools cannot execute a model outside macOS. They are conversion-verified and precision-verified only.

License

Shared freely for the sake of Allah, under free / non-commercial terms, not Apache-2.0. Access is gated: free apps only, no sales, no paid subscriptions or paywalls, no ad revenue, no monetization of any kind, and you pass these terms on. See LICENSE .