eulogik / polywhisper

PolyWhisper: Per-Language LoRA Experts on Shared Whisper-Base

5 languages · 6 experts · 1 frozen 74M encoder · 0 encoder retraining

PolyWhisper is a family of tiny language-specific LoRA experts on a frozen Whisper-Base encoder. It tackles a problem vanilla Whisper-Base cannot even begin to solve: producing correct-script text for Indian languages .

The script-confusion finding

Vanilla Whisper-Base doesn't just perform worse on Indic languages — it emits the wrong script entirely (Urdu-Arabic for Telugu/Bengali/Marathi, 0–0.5% correct-script rate). The same encoder produces near-perfect Devanagari for Hindi but collapses to Arabic for the other Indic scripts.

Per-language LoRA experts on the shared frozen encoder fix this completely.

Architecture

Audio (16kHz) → Whisper-Base Encoder [frozen, 74M]
                         ↓
          Per-language LoRA adapter [rank-16, ~0.8M each]
                         ↓
          Transcription (correct-script, correct language)

• Backbone : openai/whisper-base — frozen encoder + frozen base decoder
• Experts : rank-16 LoRA on decoder cross-attention K/V projections
• Router (Hinglish only) : 33K-param MLP on decoder hidden states, per-token en/hi selection

Results

Hinglish code-switch (v5 router, 3,129 utterances)

Indic experts (FLEURS, ortho-normalized, script-matched, 256 decode tokens)

Vanilla scores for te/bn/mr are unreportable (0–0.5% correct-script output). Experts restore correct-script and cut CER 10–22 points. Residual WER is the 39M-parameter ceiling.

Files

Hinglish router

Indic experts

Evaluation

Usage (Indic expert example)

import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration
from peft import PeftModel

proc = WhisperProcessor.from_pretrained("openai/whisper-base")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-base")
model = PeftModel.from_pretrained(model, "eulogik/polywhisper", "ta_best_ta.pt")
model.eval()

audio = ...  # 16kHz mono
feats = proc.feature_extractor([audio], sampling_rate=16000, return_tensors="pt").input_features
out = model.generate(feats, max_new_tokens=256, language="ta", task="transcribe")
print(proc.decode(out[0], skip_special_tokens=True))

Training

• 3 epochs, LR 1e-4, rank-16 LoRA on decoder cross-attention K/V
• Data: IndicVoices-ST (~19–20K samples, gated) + FLEURS (test)
• Trained on M4 Mac Mini 16GB (Apple MPS, fp16)
• Training time: ~3h per language, fully automated ( train_v3.py --langs ta )

Key finding for the paper

Language-specific LoRA experts on a shared frozen encoder fix Whisper-Base's script-confusion problem for 5 Indian languages. Vanilla outputs Urdu-Arabic for Telugu/Bengali/Marathi (0% correct-script); experts restore 77–100% correct-script output. Hindi expert reaches 38.5% WER / 14.4% CER — near-usable for edge deployment.

License

MIT. © 2026 Eulogik . Whisper is OpenAI's model. Data from IndicVoices-ST (CC-BY) and FLEURS (CC-BY 4.0).

Citation

@misc{eulogik2026polywhisper,
  title={PolyWhisper: Per-Language LoRA Experts on Shared Encoder for Indic ASR},
  author={Eulogik},
  year={2026},
  howpublished={\url{https://huggingface.co/eulogik/polywhisper}},
  note={MIT licensed; 5 Indian languages + Hinglish router; script-confusion fix}
}