mispeech / midashenglm-7b-fp8

This repository contains the fp8 quantized weights of the original model, which provides substantial memory savings and faster inference throughput while retaining overall task performance close to the bf16 release . As quantization introduces numerical approximations, individual outputs may differ slightly from the full-precision model. If you need maximum numerical fidelity (e.g., strict reproduction), use the fp32 model .

🔥 Key Highlights

State-of-the-Art Performance

• Outperforms Qwen2.5-Omni-7B, Kimi-Audio-Instruct-7B on multiple key audio understanding tasks .

High Efficiency

• 3.2× throughput speedup at comparable batch sizes compared to Qwen2.5-Omni-7B.
• 20x throughput speedup by increasing furhter batchsizes. We tested up to a batch size=512 for 30s audio input on 80GB GPUs. Baselines only support batch size = 8.
• Time-to-first-token (TTFT) speedup of up to 4x compared to Qwen2.5-Omni-7B.

Caption-based Alignment

• Trained with general audio captions (instead of ASR transcripts) to achieve holistic audio understanding.

Full Transparency

• Public-source training data and reproducible pipeline.
• Apache License 2.0 for both research and commercial use .

Acknowledgment and Model Foundation

Although MiDashengLM demonstrates superior audio understanding performance and efficiency compared to Qwen2.5-Omni models, we acknowledge Qwen2.5-Omni as a remarkable and respected foundational work in the field. Our model specifically uses Qwen2.5-Omni-7B Thinker as the initialization for decoder training, building upon its robust architecture and weight initialization.

The audio encoder is built upon Dasheng , an open-source audio encoder for general audio understanding with state-of-the-art performance. Dasheng serves as the core foundation enabling MiDashengLM's exceptional performance .

Framework

MiDashengLM integrates the powerful Dasheng audio encoder with the Qwen2.5-Omni-7B Thinker decoder through a unique caption-based alignment strategy. Unlike conventional ASR-driven approaches, our model leverages general audio captions to capture comprehensive audio representations encompassing speech, environmental sounds, and musical elements in a unified textual format. This design enables holistic audio understanding while maintaining exceptional computational efficiency.

Why Captions Instead of ASR?

ASR Limitations:

• Discards huge amount of non-speech audio (music/environmental sounds).
• Misses paralinguistic info (speaker emotion, acoustic properties).
• Monotonic alignment provides trivial learning signal.

Caption Advantages:

• Utilizes all audio content.
• Captures global audio context.
• Non-monotonic alignment provides a hard learning signal.

Novel Open Source Dataset for Training: ACAVCaps

ACAVCaps is a meticulously curated 38,662-hour collection of general audio captions derived from the open-source ACAV100M audio repository . While leveraging ACAV100M's extensive raw audio materials, we completely re-engineered the annotation process to create a dataset for holistic audio understanding. We devide the dataset into six categories:

The figure below illustrates our data curation pipeline for ACAVCaps:

Each caption is generated through a three-step process:

1. Multi-expert analysis (speech, vocal, music, acoustics)
2. LLM reasoning synthesizing metadata with DeepSeek-R1
3. Filtering for audio-text consistency with Dasheng-GLAP

We will release the ACAVCaps dataset after the ICASSP 2026 review process.

Usage

Load Model

from transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer

model_id = "mispeech/midashenglm-7b-fp8"

model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

Construct Prompt

user_prompt = "Caption the audio."  # You may try any other prompt

messages = [
    {
        "role": "system",
        "content": [
            {"type": "text", "text": "You are a helpful language and speech assistant."}
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "text", "text": user_prompt},
            {
                "type": "audio",
                "path": "/path/to/example.wav",
                # or "url": "https://example.com/example.wav"
                # or "audio": np.random.randn(16000)
            },
        ],
    },
]

Generate Output

import torch

with torch.no_grad():
    model_inputs = processor.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=True,
        add_special_tokens=True,
        return_dict=True,
    ).to(device=model.device, dtype=model.dtype)
    generation = model.generate(**model_inputs)
    output = tokenizer.batch_decode(generation, skip_special_tokens=True)  # ["An engine is idling."]

Results

MiDashengLM delivers solid performance across diverse audio understanding tasks.

Audio Captioning Results

Metrics: FENSE (higher is better).

Audio and Paralinguistic Classification

ASR Performance

Metrics: WER/CER (lower is better).

Question Answering Results

Metrics: Higher is better.

Reproduction Instructions

To reproduce our results, we provide:

• Prompts ( prompt.csv )
• Evaluation scripts
• Example JSONL files

1. Install Dependencies for Evaluation (No need this for inference)

pip install -r requirements.txt

2. Generate Model Outputs

Generate responses using the model's official framework with prompts from prompt.csv .

3. Convert Outputs to JSONL Format

Format model outputs using the example JSONL files:

4. Evaluate Results

Execute the corresponding evaluation scripts:

# Automatic Speech Recognition (WER)
# Uses: lang, text, model_output
python evaluate/wer/compute_wer.py -i evaluate/jsonl/MiDashengLM_LibriSpeech_test-clean.jsonl

# Single-target Audio Tagging (ACC)
# Uses: label, model_output
python evaluate/compute_at_acc.py -i evaluate/jsonl/MiDashengLM_NSynth.jsonl

# Gender Recognition (ACC)
# Uses: label, model_output
python evaluate/compute_gender_acc.py -i evaluate/jsonl/MiDashengLM_VoxCeleb-Gender.jsonl

# Multi-target Audio Tagging (mAP)
# Uses: dataset_name, label, model_output, model_name
python evaluate/compute_map.py -i evaluate/jsonl/MiDashengLM_FSD50K.jsonl

# Audio Captioning (FENSE)
# Uses: audio, text, model_output
python evaluate/compute_fense.py -i evaluate/jsonl/MiDashengLM_AutoACD.jsonl

# Open Audio QA (FENSE)
# Uses: audio, answer, model_output
python evaluate/compute_fense.py -i evaluate/jsonl/MiDashengLM_MusicQA.jsonl

# Audio QA with Options (ACC)
# Uses: answer, model_output
python evaluate/compute_qa_acc.py -i evaluate/jsonl/MiDashengLM_MuChoMusic.jsonl

5. Evaluate on MECAT and MMAU benchmarks

Please refer to the official repositories for evaluation on the MECAT and MMAU benchmarks.

Efficiency

MiDashengLM demonstrates superior inference efficiency compared to Qwen2.5-Omni-7B, achieving 3.2× speedup at comparable batch sizes and an overall potential speedup of 20.2× with larger batches.

Tested on 80GB GPU with 30s audio, 100-token output.

Training Data

MiDashengLM is trained exclusively on publicly available datasets across five categories: Speech, Sound and General Audio, Speech and Paralinguistic, Music, and Question Answering. All datasets are listed below with their respective tasks, lengths, and supervised fine-tuning (SFT) usage.

Speech Training Data

This table lists speech-related datasets used for tasks like Automatic Speech Recognition (ASR), keyword spotting (KWS), and speech-to-text translation (S2TT). The column “SFT?” indicates whether the dataset is used for supervised fine-tuning.

Sound and General Audio Datasets

Speech and Paralinguistic Datasets

Music-Related Datasets

Covers music captioning, genre recognition, instrument classification, and singing style identification.

Question Answering Datasets

Used for training on audio-visual QA, environment QA, and music QA tasks. Most support SFT.

Citation

MiDashengLM is under the Apache License 2.0, and we encourage its use in both research and business applications .

If you find MiDashengLM useful in your research, please consider citing our work:

@techreport{midashenglm7b,
  title      = {MiDashengLM: Efficient Audio Understanding with General Audio Captions},
  author     = {{Horizon Team, MiLM Plus}},
  institution= {Xiaomi Inc.},
  year       = {2025},
  note       = {Contributors: Heinrich Dinkel et al. (listed alphabetically in Appendix B)},
  url        = {https://arxiv.org/abs/2508.03983},
  eprint     = {2508.03983},
}