igorktech / femto-asr

femto-asr

2.39M parameters. 9.2 MB in fp32, 2.5 MB as int8. An English speech recognizer sized for a microcontroller with 400 KB of SRAM and no floating-point unit.

It will not beat Whisper. It is for the case where the alternative is no speech recognition at all.

Results

LibriSpeech, greedy CTC, no language model, no rescoring, full utterances:

For command-and-control use, a lexicon-constrained decoder over a known phrase list takes error rates far lower (~0.28 → ~0.05 WER on a 198-utterance phrase set) — but that is a decoder choice, not this model.

Architecture

 audio 16 kHz mono
      │
      ▼
 log-mel 80 bins @ 100 fps ── global CMVN
      │
      ▼
┌─────────────────────────────────────────────┐
│ conv stem            80→128, k5 s2          │   4x downsample
│                     128→128, k5 s1          │   100 fps → 25 fps
│                     depthwise k5 s2 + 1x1   │
└─────────────────────────────────────────────┘
      │  [B, T, 128]
      ▼
╔═════════════════════════════════════════════╗
║  E-Branchformer block   × 12                ║
║                                             ║
║   ┌── LN → linear GQA-latent attention ──┐  ║  branch 0
║   │                                      ├──╫─► w₀·attn + w₁·cgmlp
║   └── LN → conv-gating MLP (k31) ────────┘  ║     (learned 2-vector)
║                                             ║
║       LN → SwiGLU FFN, rank-64 ─────────────╫─► branch 1
╚═════════════════════════════════════════════╝
      │                    ▲
      │   each branch reads a learned mix of earlier
      │   residual snapshots, not just the running sum
      │   (depth-wise linear attention, 24 steps)
      ▼
 linear head → 1025 CTC classes  (0 = blank)

Per block: 4 query heads over 2 KV groups, both reached through 16-wide latents; SwiGLU hidden width 512 factored to rank 64; depthwise kernel 31.

Four choices carry the model, and each is load-bearing:

• Linear attention, not softmax. Keys and values collapse into one [32, 32] state per group, so cost is linear in time and no T×T matrix ever exists — which is what makes 400 KB of SRAM feasible. The feature map is relu² rather than elu + 1 because an exp is a soft-float call per element on a chip with no FPU.
• Low-rank FFN. All three SwiGLU projections are rank-64 factorizations. Training the same model at rank 32 costs ~11% relative WER, and truncating a trained rank-64 to 48 roughly triples it.
• NoRA ( arXiv:2608.31036 ): each low-rank down-projection is L2-normalized along the rank dimension. ~9% relative WER for no parameters and no compute — it folds into the weights at export.
• Fixed-point RMSNorm. The depth mixer rounds its rsqrt scale to 16 fractional bits, so an integer Newton-Raphson rsqrt on-device reproduces training numerics exactly. Active at inference, not a training-only trick.

Training

960 hours of LibriSpeech, 35 epochs, one RTX 4090, 16 hours.

Trained from scratch with CR-CTC consistency regularization, SpecAugment on a ramp, an auxiliary CTC head at an intermediate layer (dropped at export), the Muon optimizer, a Noam schedule with a terminal linear cooldown, and EMA weights — all reported numbers are the EMA. Speed perturbation between 0.85x and 1.3x was applied from the first step rather than bolted on afterwards; that ordering matters (see Speaking rate ).

Training loss rises between epochs 7 and 14 (79 → 118) while validation keeps improving. That is the SpecAugment ramp increasing augmentation strength, not divergence — the model is being shown harder inputs, so its loss on them grows even as it generalizes better. Validation WER is the curve to read.

The last 7 epochs of cooldown are worth ~3% relative, and the final checkpoint beat the best-validation one (epoch 32) on all four splits, so epoch 35 is what ships.

Usage

import soundfile as sf
from transformers import AutoModelForCTC, AutoFeatureExtractor, AutoTokenizer

REPO = "igorktech/femto-asr"
model = AutoModelForCTC.from_pretrained(REPO, trust_remote_code=True).eval()
fe    = AutoFeatureExtractor.from_pretrained(REPO, trust_remote_code=True)
tok   = AutoTokenizer.from_pretrained(REPO, trust_remote_code=True)

wav, sr = sf.read("clip.wav")               # 16 kHz mono
out = model(**fe(wav, sampling_rate=sr))
ids = model.greedy_decode(out.logits, out.output_lengths)
print(tok.decode(ids[0]))

With the pipeline:

from transformers import pipeline
pipe = pipeline("automatic-speech-recognition", model=REPO, trust_remote_code=True)
print(pipe("clip.wav")["text"])

Batching — ragged inputs are padded and masked, and batched results match single-clip results exactly:

batch = fe([wav1, wav2, wav3], sampling_rate=16000)
out = model(**batch)
texts = [tok.decode(x) for x in model.greedy_decode(out.logits, out.output_lengths)]

Fine-tuning — pass labels as CTC class indices and you get a CTC loss back:

labels = torch.tensor([tok.encode_text("HELLO WORLD")])   # already class indices
loss = model(**fe(wav, sampling_rate=16000), labels=labels).loss
loss.backward()

trust_remote_code=True is required: this is a custom architecture, so the modeling code lives in the repo.

Notes for anyone modifying this repo

Padding. The feature extractor adds 0.3 s of silence to both ends. Frames at either edge have context on one side only: without the leading pad the first word is frequently dropped, without the trailing pad the last token truncates. On a live device, start the encoder ~0.3 s before the button and keep it running ~0.3 s after release.

tokenizer_class must stay out of tokenizer_config.json . In transformers 5.x, AutoTokenizer.from_pretrained returns early when that key is present and never consults auto_map , which breaks custom-code tokenizers with a confusing NoneType has no attribute from_pretrained . save_pretrained will put the key back — remove it again, or loading fails.

Id convention. Tokenizer ids are CTC class indices (0 = blank, BPE token i at i + 1 ), following Wav2Vec2 so the pipeline works unmodified. decode does the full CTC reduction. Use encode_bpe_ids() if you need raw BPE ids for the C implementation.

Limitations

• English read speech. Trained on LibriSpeech audiobooks. Accents, spontaneous and conversational speech, children's voices, and far-field or noisy audio are all out of distribution. LibriSpeech is not demographically representative; evaluate on your own audio and speakers before relying on this.
• Fast speech degrades sharply. See below.
• No language model. Greedy CTC gets homophones and rare words wrong in ways an LM or a constrained decoder would fix. Proper nouns are particularly weak — it will not know a name it never saw.
• Uppercase only , no punctuation, no casing, no digits.
• 16 kHz mono only.
• Not streaming as published. Attention is full-context. The architecture supports chunk-causal operation but these weights were trained and evaluated offline.
• Sensitive to spectral smearing. Reverberant rooms are a plausible failure mode; a phase-vocoder artifact alone cost 2.5x WER in testing.

Speaking rate

dev-clean, time-compressed with a pitch-preserving phase vocoder:

Training with perturbation from step one beat adapting a converged model afterwards at every rate including 1.0x, whereas the retrofit bought fast-speech robustness by paying 2–4% at normal speed.

If you benchmark this yourself: librosa.effects.time_stretch at its defaults uses a 2048-sample window, which is 128 ms at 16 kHz and smears speech transients badly. That alone cost 2.5x WER and made fast speech look far worse than it is. Use a shorter window.

Licence

Apache 2.0, covering the code and the model weights.

Two things it does not cover. The logo depicts a character owned by its respective rights holders and is not part of the licence grant. And the model was trained on LibriSpeech, which is CC BY 4.0 — credited below, and not a restriction on use of these weights.

Credits

Combines E-Branchformer (Kim et al., 2022), MLA-style latent attention (DeepSeek-V2), grouped-query attention, linear attention with a positive feature map, Fast Conformer's depthwise-separable subsampling ( arXiv:2305.05084 ), CR-CTC, and NoRA ( arXiv:2608.31036 ).