eulogik / polywhisper-hinglish-router

PolyWhisper Hinglish Router

A code-switch (Hindi–English) speech recognition model by Eulogik — frozen Whisper-Base encoder + two rank-8 LoRA language experts + a 33K-parameter per-token router.

This is the v5 release. The router was trained with corrected per-token language labels, and it is the first PolyWhisper checkpoint where per-token routing is genuinely learned and measured.

Why it matters

Speakers in India switch between Hindi and English mid-sentence (Hinglish). Single-language ASR models degrade on this. PolyWhisper:

• Routes every token to an English-expert or Hindi-expert LoRA,
• Adds +13.3 WER points over a static 50/50 expert mix (58.8% vs 72.1%),
• Adds +7.8 WER points over vanilla Whisper-Base (66.6%),
• Cuts hallucinated repetition loops ~20× (13 vs 279 events),
• Trains in ~a day on a 16GB Apple Silicon Mac — no GPU cluster.

Results (3,129-utterance code-switched test set)

Router per-token language accuracy: 89.1% (99,663 / 111,815 tokens).

Files

Usage

import torch, soundfile as sf
from transformers import WhisperProcessor
from model import PolyWhisperRouter   # see github.com/eulogik/PolyWhisper

model = PolyWhisperRouter().to("mps" if torch.backends.mps.is_available() else "cpu")
model.add_language("en").add_language("hi")
model.load_adapter("en", "en_router_best_v5.pt")
model.load_adapter("hi", "hi_router_best_v5.pt")
model.load_router("router_best_v5.pt")

audio, _ = sf.read("hinglish.wav")
feats = WhisperProcessor.from_pretrained("openai/whisper-base").feature_extractor(
    [audio], sampling_rate=16000, return_tensors="pt").input_features
out = model.generate(feats, max_new_tokens=128, use_cache=False, language="hi", task="transcribe")

Training

• Backbone: openai/whisper-base (frozen encoder + frozen base decoder)
• Experts: rank-8 LoRA on decoder cross-attention (K/V) — ~3M params each
• Router: 2-layer MLP over decoder hidden states (33K params)
• Stage A1: EN expert, masked to English tokens (3 epochs, ~9h on M4 MPS)
• Stage A2: HI expert + encoder-LoRA (4 epochs, ~9h)
• Stage P1: router-only language selection (2 epochs)
• Stage P2: joint router + expert adaptation (2 epochs, λ_router=1.0)
• Data: MUCS / IndicVoices-ST derived code-switched Hinglish (~42K train)

Limitations

• Word error rate is high in absolute terms (~58.8%) — acceptable for edge/low-resource use, not parity with large models
• Tested on Hinglish tutorial-style speech; robustness to spontaneous/overlapping speech untested
• Devanagari orthography variance still inflates WER (see per-sample errors)
• No privacy guarantees; data is public corpora

License

MIT. © 2026 Eulogik . Whisper is OpenAI's model. Derived data from MUCS (CC-BY-SA) and IndicVoices-ST (CC-BY) — see the GitHub repo for attribution.

Citation

@misc{eulogik2026polywhisper,
  title={PolyWhisper: Code-Switch ASR with Per-Token LoRA Routing for Hinglish},
  author={Eulogik},
  year={2026},
  howpublished={\url{https://huggingface.co/eulogik/polywhisper-hinglish-router}},
  note={MIT licensed; benchmarked on 3,129-utterance code-switched test set}
}