nyralabs / nyra_forced_aligner_en

nyra-forced-aligner

The most accurate word timing you can get for conversational speech: verbatim-aware, noise-robust, and fast.

Blog post · Paper · Models · GitHub · CrisperWhisper

A forced aligner answers one question: given a recording and its transcript, when was each word said? Most aligners were built for read speech and clean audio, and they quietly fall apart on the material people actually need to align: conversations full of um s, laughter, restarts and half-finished words, recorded in rooms that are not studios.

nyra-forced-aligner was built for exactly that material.

• Vocal sounds are alignable. Fillers ( [UH] , [UM] ), laughter, breaths, coughs and word cut-offs ( th- ) are units of the model like any phone, so they get timestamps of their own instead of being smeared into the neighbouring words.
• Made for CrisperWhisper transcripts. Trained on the verbatim transcription convention of CrisperWhisper 2.0 : feed its output straight in and every filler, repetition and event token is aligned.
• Accurate where it matters. 19.9 ms mean word-boundary error on conversational English (Buckeye): 21% lower than the best configuration of the Montreal Forced Aligner, half the error of the best ASR-with-timestamps system, and the only system whose error tail drops below 1% of words before 100 ms.
• Noise-robust. Every utterance stays aligned down to 0 dB SNR, where classical aligners lose or mangle most of them. A Pro model goes further for difficult recordings.
• Fast and longform. Batched WavLM features, GPU emissions and a compiled beam Viterbi: eight minutes of audio align in about three seconds on one GPU.

Install

pip install nyra-forced-aligner

# espeak-ng is needed to pronounce words outside the built-in 41k-word lexicon:
#   apt install espeak-ng      (macOS: brew install espeak-ng)

Runs on CPU, but a CUDA GPU is strongly recommended (WavLM-large front-end). The model is downloaded from the Hugging Face Hub on first use.

Quickstart

from nyra_align import Aligner

aligner = Aligner()                      # nyralabs/nyra_forced_aligner_en
# aligner = Aligner(pro=True)            # noise-robust model

result = aligner.align("interview.wav",
                       "so i [UM] i went there on on thursday [laughter]")

for w in result.words:
    print(f"{w.start:7.3f} {w.end:7.3f}  {w.word:<12s} {w.type}")   # w / f(iller) / s(ound) / c(ut-off)

result.to_textgrid("interview.TextGrid")     # Praat
result.to_json("interview.json")

Any audio length works. For the best timings follow the transcript conventions below.

Command line, single file or a whole corpus of paired .wav + .txt / .lab files (MFA-style layout):

nyra-align interview.wav --text-file interview.txt --out interview.TextGrid
nyra-align corpus_dir/ --pro --out-dir aligned/ --format textgrid

Transcript conventions

The aligner can only place what the transcript contains, so for the best timings the transcript should be verbatim: write what you hear.

• Cut-offs. Interrupted words and word fragments are marked with a trailing hyphen: th- , w- , resched- .
• Fillers. Filled pauses are bracketed and upper-cased: [UH] , [UM] .
• Vocal sound events. Non-speech sounds are bracketed: [laughter] , [cough] , [breath] , [sigh] , [sniff] , [lipsmack] , [throatclearing] , [yawn] , [noise] .
• Numbers, dates, times and emails are written out exactly as spoken ( March third at nine thirty ), not as digits or symbols.
• Everything else is transcribed word for word. Normal casing and punctuation are fine (the aligner ignores both); repetitions, false starts, fillers, fragments and sounds are all left in place.

so we we need to, to reschedule the th- Thursday meeting to [UH] March third at nine thirty [laughter]

What happens when a transcript deviates: a word that was spoken but is missing from the transcript gets absorbed into its neighbours, and a word in the transcript that was never spoken is squeezed into a few frames, so both shift the surrounding boundaries. Event tags are matched in any casing; a tag the model does not know ( [music] ) is left out and listed in result.skipped , as is any token that cannot be pronounced.

With CrisperWhisper 2.0

CrisperWhisper 2.0 produces transcripts in exactly this convention, so its verbatim output can be aligned as is:

from crisperwhisper import CrisperWhisperModel
from nyra_align import Aligner

audio = "interview.wav"
transcript = CrisperWhisperModel("large").transcribe(audio, language="en").text
result = Aligner().align(audio, transcript)

for w in result.words:
    print(f"{w.start:7.3f} {w.end:7.3f}  {w.word}")
result.to_textgrid("interview.TextGrid")

Models

Only English is available at the moment; Aligner(language="de") raises NotImplementedError .

Performance

Mean absolute word-boundary error on Buckeye (conversational English, 2,008 test recordings, 80,475 words), lower is better. Every system is scored with the same protocol: DP word matching on canonicalized forms, laughter tokens excluded, MAE = mean over words of (|onset error| + |offset error|)/2.

_{MAE is over the per-word error (|onset error| + |offset error|)/2. The ≤ t ms columns follow the convention of Rousso et al. (2024): the share of words whose end timestamp is within t of the reference (one boundary, not the per-word mean). Forced aligners are given the reference words and only predict their timing; every word is scored. ASR systems produce their own words, which are text-aligned to the reference, and only matched words are scored (88–93% of words), so their timing is judged independently of their transcription errors. ¹ MFA's shipped default beam (10) prunes the correct path on conversational speech; the "wide beam" rows use beam 400 / retry 4000 , the best configuration we found for it.}

The mean hides where the difference really is. Averaged over all words, most systems are within a few tens of milliseconds; the gap is in the tail , the words that are misaligned by 100 ms or more, which is what breaks downstream use. Each curve shows the share of words whose per-word error (mean of the onset and offset errors, the quantity behind MAE) is larger than t ; note this is not the single-boundary ≤ t ms column of the table:

Under noise

The same Buckeye recordings with added noise, at signal-to-noise ratios from 20 dB down to 0 dB (MAE in ms). MUSAN is real noise recordings; Gaussian is white noise, which the model never saw in training.

_{² CrisperWhisper 2.0 is an ASR system: at 0 dB it still recognises only 29% of the words, and only those are scored. All forced aligners above keep 100% of words at every SNR.}

nyra_forced_aligner_en_pro delivers roughly the same accuracy on clean speech and is meaningfully more robust to noise and adverse acoustic conditions.

Results on read speech (TIMIT) and stuttered speech (FluencyBank), the full set of ablations, and how the models are trained are in the paper.

What else is in the box

How it works

The full story is in the deep-dive post:

• The nyra forced aligner: precise word timing for real conversational speech . Why classical aligners break on spontaneous and noisy speech, how frozen WavLM features, vocal-event units and a noise-pooled projection fix it, and how the model bootstraps itself from verbatim transcripts without any manual boundary labels.

In short: WavLM-large hidden states are projected by a supervised LDA onto 40 dimensions and modelled by a triphone GMM-HMM whose inventory holds 15 vocal-event units next to the 39 English phones. At inference the package builds the alignment graph for the transcript (optional silence between words, full context across word boundaries) and runs a compiled beam Viterbi over the model's likelihoods, with a beam ladder so that no utterance is ever silently dropped. Words outside the built-in lexicon are phonemized with espeak through the same mapping used at training time.

License

The model files are released under the nyra health Non-Commercial Research License : free for research and other non-commercial use. This license covers the model and the alignments it produces : any commercial use of the timestamps, and any use of them to train, fine-tune or otherwise improve models intended for commercial use, requires a commercial license. The Pro model is available under commercial license only. For commercial licensing of either, contact nyra health .

The model repositories bundle Microsoft's WavLM-large weights, redistributed unchanged under their upstream license.