mispeech / midashenglm-gen

English | 中文

MiDashengLM-Gen is an end-to-end framework that uses a pre-trained Large Language Model and audio tokenizer as the backbone, combined with per-token conditional flow matching for autoregressive, variable-length mixed-audio scene generation . It generates coherent 16 kHz audio scenes that simultaneously blend speech, music, sound effects, and environmental acoustics from structured text descriptions.

Architecture

Left: training pipeline with flow matching loss. Right: autoregressive inference pipeline.

Key components:

Input Format

Input uses structured multi-view captions with special tokens to describe different aspects of an audio scene. Use <|unknown|> for absent elements.

Installation

pip install torch torchaudio "transformers>=4.51" einops safetensors soundfile tqdm numpy x-transformers

Quick Start

from transformers import AutoModel
import soundfile as sf

model = AutoModel.from_pretrained("mispeech/midashenglm-gen", trust_remote_code=True)
model = model.cuda()

result = model.generate(
    "<|caption|> A comedian delivering a punchline followed by uproarious crowd laughter and an upbeat jazz band hit "
    "<|asr|> And that is why I never buy cheap luggage anymore! "
    "<|speech|> expressive comedic male voice "
    "<|music|> sudden upbeat jazz band sting "
    "<|sfx|> uproarious crowd laughter "
    "<|env|> intimate comedy club"
)

sf.write("output.wav", result["audio"], result["sample_rate"])

Batch Generation

texts = [
    "<|caption|> A comedian delivering a punchline followed by uproarious crowd laughter and an upbeat jazz band hit <|asr|> And that is why I never buy cheap luggage anymore! <|speech|> expressive comedic male voice <|music|> sudden upbeat jazz band sting <|sfx|> uproarious crowd laughter <|env|> intimate comedy club",
    "<|caption|> Upbeat instrumental music with occasional whistling and low audio quality. <|asr|> <|unknown|> <|speech|> <|unknown|> <|sfx|> Intermittent whistling emerges in later segment alongside persistent instrumental playback. <|music|> Bright melodic composition with guitar, piano, and rhythmic percussion elements, maintaining a consistently upbeat character. <|env|> Low-fidelity recording with compressed dynamics and limited frequency response.",
]
result = model.generate(texts)
for i, audio in enumerate(result["audio"]):
    sf.write(f"output_{i}.wav", audio, result["sample_rate"])

Generation Parameters

result = model.generate(
    "...",
    eval_cfg=2.0,           # Classifier-free guidance scale
    stop_threshold=0.5,     # Stop prediction threshold
    min_stop_step=5,        # Minimum steps before stopping
    seed=42,                # Random seed for reproducibility
)

Citation

@article{sun2026midashenglmgen,
  title={MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching},
  author={Sun, Xingwei and Dinkel, Heinrich and Li, Gang and Mei, Jiahao and Niu, Yadong and Han, Zerui and Jiang, Yuepeng and Zhou, Jiahao and Fan, Lichun and Luan, Jian},
  journal={arXiv preprint},
  eprint={2608.11804},
  archivePrefix={arXiv},
  year={2026}
}

License

Apache 2.0

You are solely responsible for your use of MiDashengLM-Gen and any outputs, actions, or consequences arising therefrom, and you agree not to use MiDashengLM-Gen or any derivatives thereof:

• For any unlawful, fraudulent, or malicious purpose, or in any manner that violates any applicable laws or regulations;
• To infringe upon the intellectual property rights, privacy rights, publicity rights, or other lawful rights or interests of any third party;
• To exploit, harm, harass, defame, unlawfully discriminate against, or otherwise adversely affect any individual or group, including minors or vulnerable persons;
• For any military purpose or application.