ASLP-lab / Speaker-Reasoner

Speaker-Reasoner: Scaling Interaction Turns and Reasoning Patterns for Timestamped Speaker-Attributed ASR

Speaker-Reasoner is an end-to-end Speech LLM for timestamped speaker-attributed ASR featuring agentic multi-turn temporal reasoning. Instead of single-pass inference, the model iteratively analyzes global audio structure, autonomously predicts temporal boundaries, and performs fine-grained segment analysis, jointly modeling speaker identity, gender, timestamps, and transcription. A speaker-aware cache further extends processing to audio exceeding the training context window.

🌟 Highlights

• Agentic multi-turn reasoning : iterative global-to-local inference along the temporal axis — global speaker summary → boundary prediction → fine-grained segment decoding
• Speaker-aware context cache : extends processing to long-form audio beyond the training context window while preserving speaker consistency across chunks
• Three-stage progressive training : multi-task foundation → temporal interaction learning → cache-conditioned decoding
• State-of-the-art performance : outperforms strong baselines including closed-source Gemini-2.5-Pro on AliMeeting and AISHELL-4
• 🔥 Bilingual & Scaled up : extended training on 4,194 hours of multi-domain data, natively supporting English and Mandarin across complex multi-speaker scenarios

📊 Results

Comprehensive Multi-Domain Evaluation

We further scaled up Speaker-Reasoner with 4,194 hours of bilingual (ZH/EN) training data. The model demonstrates superior performance across diverse scenarios, including challenging video domains and various public meeting datasets.

Segmented Evaluation (40–50s segments)

† Closed-source model. DER unavailable for SpeakerLM and SOT-based models due to incompatible output formats.

Long-form Evaluation (without segmentation)

Speaker Attribute Evaluation (AISHELL4-Eval)

Installation

Environment Setup

git clone https://github.com/ASLP-lab/Speaker-Reasoner.git
cd Speaker-Reasoner

conda create -n speaker-reasoner python=3.10 -y
conda activate speaker-reasoner

Install MS-Swift and dependencies:

pip install ms-swift

Model Download

We provide the pre-trained model weights on Hugging Face. You can download the corresponding versions based on your requirements:

Training

Coming soon.

Inference

vLLM

Speaker-Reasoner is built on top of Qwen3-Omni-30B-A3B-Instruct . To run it, you will need to install a custom branch of vLLM from source.

git clone -b qwen3_omni https://github.com/wangxiongts/vllm.git
cd vllm
pip install -r requirements/build.txt
pip install -r requirements/cuda.txt
export VLLM_PRECOMPILED_WHEEL_LOCATION=https://wheels.vllm.ai/a5dd03c1ebc5e4f56f3c9d3dc0436e9c582c978f/vllm-0.9.2-cp38-abi3-manylinux1_x86_64.whl
VLLM_USE_PRECOMPILED=1 pip install -e . -v --no-build-isolation
# If you meet an "Undefined symbol" error while using VLLM_USE_PRECOMPILED=1, please use "pip install -e . -v" to build from source.
# Install the Transformers
pip install git+https://github.com/huggingface/transformers
pip install accelerate
pip install qwen-omni-utils -U
pip install -U flash-attn --no-build-isolation

For more details on compiling vLLM from source, refer to the vLLM official documentation .

Citation

If you find this work useful, please cite:

@article{lin2026speakerreasoner,
  title={Speaker-Reasoner: Scaling Interaction Turns and Reasoning Patterns for Timestamped Speaker-Attributed ASR}, 
  author={Zhennan Lin and Shuai Wang and Zhaokai Sun and Pengyuan Xie and Chuan Xie and Jie Liu and Qiang Zhang and Lei Xie},
  year={2026},
  eprint={2604.03074},
  archivePrefix={arXiv},
  primaryClass={eess.AS},
  url={https://arxiv.org/abs/2604.03074}, 
}

License

The code in this repository is released under the Apache 2.0 License .

Contact

• Issues : Please open a GitHub Issue for bug reports or suggestions.
• Email : [email protected] , [email protected]