mispeech / midashenglm-7b

🔥 Key Highlights

State-of-the-Art Performance

• Outperforms Qwen2.5-Omni-7B, Kimi-Audio-Instruct-7B on multiple key audio understanding tasks .

High Efficiency

• 3.2× throughput speedup at comparable batch sizes compared to Qwen2.5-Omni-7B.
• 20x throughput speedup by increasing furhter batchsizes. We tested up to a batch size=512 for 30s audio input on 80GB GPUs. Baselines only support batch size = 8.
• Time-to-first-token (TTFT) speedup of up to 4x compared to Qwen2.5-Omni-7B.

Caption-based Alignment

• Trained with general audio captions (instead of ASR transcripts) to achieve holistic audio understanding.

Full Transparency

• Public-source training data and reproducible pipeline.
• Apache License 2.0 for both research and commercial use .

Acknowledgment and Model Foundation

Although MiDashengLM demonstrates superior audio understanding performance and efficiency compared to Qwen2.5-Omni models, we acknowledge Qwen2.5-Omni as a remarkable and respected foundational work in the field. Our model specifically uses Qwen2.5-Omni-7B Thinker as the decoder starting point during training, building upon its robust architecture and weight initialization.

The audio encoder is built upon Dasheng , an open-source audio encoder for general audio understanding with state-of-the-art performance. Dasheng serves as the core foundation enabling MiDashengLM's exceptional performance .

Framework

MiDashengLM integrates the powerful Dasheng audio encoder with the Qwen2.5-Omni-7B Thinker decoder through a unique caption-based alignment strategy. Unlike conventional ASR-driven approaches, our model leverages general audio captions to capture comprehensive audio representations encompassing speech, environmental sounds, and musical elements in a unified textual format. This design enables holistic audio understanding while maintaining exceptional computational efficiency.

Why Captions Instead of ASR?

ASR Limitations:

• Discards huge amount of non-speech audio (music/environmental sounds).
• Misses paralinguistic info (speaker emotion, acoustic properties).
• Monotonic alignment provides trivial learning signal.

Caption Advantages:

• Utilizes all audio content.
• Captures global audio context.
• Non-monotonic alignment provides a hard learning signal.

Novel Open Source Dataset for Training: ACAVCaps

ACAVCaps is a meticulously curated 38,662-hour collection of general audio captions derived from the open-source ACAV100M audio repository . While leveraging ACAV100M's extensive raw audio materials, we completely re-engineered the annotation process to create a dataset for holistic audio understanding. We devide the dataset into six categories:

The figure below illustrates our data curation pipeline for ACAVCaps:

Each caption is generated through a three-step process:

1. Multi-expert analysis (speech, vocal, music, acoustics)
2. LLM reasoning synthesizing metadata with DeepSeek-R1
3. Filtering for audio-text consistency with GLAP

We will release the ACAVCaps dataset after the ICASSP 2026 review process.

Usage

Load Model

from transformers import AutoModelForCausalLM, AutoProcessor

model_id = "mispeech/midashenglm-7b"

model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

Construct Prompt

user_prompt = "Caption the audio."  # You may try any other prompt 

messages = [
    {
        "role": "system",
        "content": [
            {"type": "text", "text": "You are a helpful language and speech assistant."}
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "text", "text": user_prompt},
            {
                "type": "audio",
                "path": "/path/to/example.wav",
                # or "url": "https://example.com/example.wav"
                # or "audio": np.random.randn(16000)
            },
        ],
    },
]

Generate Output

import torch
with torch.no_grad():
    model_inputs = processor.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=True,
        add_special_tokens=True,
        return_dict=True,
    )
    generation = model.generate(**model_inputs)
    output = processor.batch_decode(generation, skip_special_tokens=True)  # ["An engine is idling."]

Results

MiDashengLM delivers solid performance across diverse audio understanding tasks.

Audio Captioning Results

Metrics: FENSE (higher is better).

Audio and Paralinguistic Classification

ASR Performance

Metrics: WER/CER (lower is better).

Question Answering Results

Metrics: Higher is better.

Reproduction Instructions

To reproduce our results, please refer to https://github.com/xiaomi-research/dasheng-lm#reproduction-instructions

Efficiency

MiDashengLM demonstrates superior inference efficiency compared to Qwen2.5-Omni-7B, achieving 3.2× speedup at comparable batch sizes and an overall potential speedup of 20.2× with larger batches.

Tested on 80GB GPU with 30s audio, 100-token output.

Citation

MiDashengLM is under the Apache License 2.0, and we encourage its use in both research and business applications .

If you find MiDashengLM useful in your research, please consider citing our work:

@misc{midashenglm7b,
    title = {MiDashengLM: Efficient Audio Understanding with General Audio Captions},
    author = {Xiaomi MiLM Plus Horizon Team},
    year = {2025},
}