espnet / tedlium2_streaming_transformer

ESPnet2 streaming ASR model

espnet/tedlium2_streaming_transformer

A streaming Transformer: it decodes while the audio is still arriving, a block at a time, rather than waiting for the end of the utterance. Trained on TED-LIUM 2 with the egs2/tedlium2/asr1 recipe and conf/train_asr_streaming_transformer.yaml , with a BPE-500 language model beside it.

Trained and originally published by Keqi Deng , as D-Keqi/espnet_asr_train_asr_streaming_transformer_raw_en_bpe500_sp_valid.acc.ave . This copy is here so that the demo notebooks and the documentation point at a model the organization keeps alive; the weights are the same ones.

Usage

import soundfile as sf
from espnet2.bin.asr_inference_streaming import Speech2TextStreaming

s2t = Speech2TextStreaming.from_pretrained(
    "espnet/tedlium2_streaming_transformer",
    device="cpu",
    beam_size=20,
    ctc_weight=0.5,
    penalty=0.0,
    nbest=1,
    disable_repetition_detection=True,
)

speech, rate = sf.read("audio.wav", dtype="float32")  # 16 kHz, one channel

# 640 samples is 40 ms: the hypothesis comes back after each slice, and grows
chunk = 640
slices = len(speech) // chunk
for i in range(slices):
    results = s2t(speech=speech[i * chunk : (i + 1) * chunk], is_final=False)
    if results:
        print(results[0][0])

results = s2t(speech=speech[slices * chunk :], is_final=True)
print(results[0][0])

There is a notebook that runs this end to end: Demos/asr_streaming_demo.ipynb .

For non-streaming recognition, OWSM-CTC is far more accurate and covers 150+ languages.

Results

TED-LIUM 2, decoded with the language model in this repository.

The full tables are in exp/asr_train_asr_streaming_transformer_raw_en_bpe500_sp/RESULTS.md .

Trained with espnet 0.9.8 and pytorch 1.5.1, in November 2021, and loaded by current ESPnet through the meta.yaml in this repository.

Citing ESPnet

@inproceedings{watanabe2018espnet,
  author={Shinji Watanabe and Takaaki Hori and Shigeki Karita and Tomoki Hayashi and Jiro Nishitoba and Yuya Unno and Nelson {Enrique Yalta Soplin} and Jahn Heymann and Matthew Wiesner and Nanxin Chen and Adithya Renduchintala and Tsubasa Ochiai},
  title={{ESPnet}: End-to-End Speech Processing Toolkit},
  year={2018},
  booktitle={Proceedings of Interspeech},
  pages={2207--2211},
  doi={10.21437/Interspeech.2018-1456},
  url={http://dx.doi.org/10.21437/Interspeech.2018-1456}
}