Sponsored by
APIMart
MBZUAI / LLMVoX
Model's Last Updated: March 16 2025
text-to-speech
Introduction of LLMVoX
Model Details of LLMVoX
This repository contains the model as described in LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM .
For more information, check out the project page at https://mbzuai-oryx.github.io/LLMVoX/ and the code at https://github.com/mbzuai-oryx/LLMVoX .
LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM
Authors:
Sambal Shikar
,
Mohammed Irfan K
,
Sahal Shaji Mullappilly
,
Fahad Khan
,
Jean Lahoud
,
Rao Muhammad Anwer
,
Salman Khan
,
Hisham Cholakkal
Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), UAE
Overview
LLMVoX is a lightweight 30M-parameter, LLM-agnostic, autoregressive streaming Text-to-Speech (TTS) system designed to convert text outputs from Large Language Models into high-fidelity streaming speech with low latency.
Key features:
- 🚀 Lightweight & Fast : Only 30M parameters with end-to-end latency as low as 300ms
- 🔌 LLM-Agnostic : Works with any LLM and Vision-Language Model without fine-tuning
- 🌊 Multi-Queue Streaming : Enables continuous, low-latency speech generation
- 🌐 Multilingual Support : Adaptable to new languages with dataset adaptation
Quick Start
Installation
# Requirements: CUDA 11.7+, Flash Attention 2.0+ compatible GPU
git clone https://github.com/mbzuai-oryx/LLMVoX.git
cd LLMVoX
conda create -n llmvox python=3.9
conda activate llmvox
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install flash-attn --no-build-isolation
pip install -r requirements.txt
# Download checkpoints from Hugging Face
# https://huggingface.co/MBZUAI/LLMVoX/tree/main
mkdir -p CHECKPOINTS
# Download wavtokenizer_large_speech_320_24k.ckpt and ckpt_english_tiny.pt
Voice Chat
# Basic usage
python streaming_server.py --chat_type voice --llm_checkpoint "meta-llama/Llama-3.1-8B-Instruct"
# With multiple GPUs
python streaming_server.py --chat_type voice --llm_checkpoint "meta-llama/Llama-3.1-8B-Instruct" \
--llm_device "cuda:0" --tts_device_1 1 --tts_device_2 2
# Balance latency/quality
python streaming_server.py --chat_type voice --llm_checkpoint "meta-llama/Llama-3.1-8B-Instruct" \
--initial_dump_size_1 10 --initial_dump_size_2 160 --max_dump_size 1280
Text Chat & Visual Speech
# Text-to-Speech
python streaming_server.py --chat_type text --llm_checkpoint "meta-llama/Llama-3.1-8B-Instruct"
# Visual Speech (Speech + Image → Speech)
python streaming_server.py --chat_type visual_speech --llm_checkpoint "Qwen/Qwen2.5-VL-7B-Instruct" \
--eos_token "<|im_end|>"
# Multimodal (support for models like Phi-4)
python streaming_server.py --chat_type multimodal --llm_checkpoint "microsoft/Phi-4-multimodal-instruct" \
--eos_token "<|end|>"
API Reference
| Endpoint | Purpose | Required Parameters |
|---|---|---|
/tts
|
Text-to-speech |
text
: String to convert
|
/voicechat
|
Voice conversations |
audio_base64
,
source_language
,
target_language
|
/multimodalchat
|
Voice + multiple images |
audio_base64
,
image_list
|
/vlmschat
|
Voice + single image |
audio_base64
,
image_base64
,
source_language
,
target_language
|
Local UI Demo
# Start server
python streaming_server.py --chat_type voice --llm_checkpoint "meta-llama/Llama-3.1-8B-Instruct" --api_port PORT
# Launch UI
python run_ui.py --ip STREAMING_SERVER_IP --port PORT
Citation
@article{shikhar2025llmvox,
title={LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM},
author={Shikhar, Sambal and Kurpath, Mohammed Irfan and Mullappilly, Sahal Shaji and Lahoud, Jean and Khan, Fahad and Anwer, Rao Muhammad and Khan, Salman and Cholakkal, Hisham},
journal={arXiv preprint arXiv:2503.04724},
year={2025}
}
Acknowledgments
License
This project is licensed under the MIT License - see the LICENSE file for details.
Runs of MBZUAI LLMVoX on huggingface.co
25
Total runs
0
24-hour runs
-1
3-day runs
6
7-day runs
17
30-day runs
More Information About LLMVoX huggingface.co Model
Url of LLMVoX
LLMVoX huggingface.co Url
Provider of LLMVoX huggingface.co
Other API from MBZUAI
Total runs:
5.2K
Run Growth:
4.8K
Growth Rate:
92.26%
Total runs:
1.7K
Run Growth:
-385
Growth Rate:
-23.05%
Total runs:
1.2K
Run Growth:
-1.1K
Growth Rate:
-91.84%
Total runs:
1.1K
Run Growth:
-98
Growth Rate:
-8.73%
Total runs:
1.1K
Run Growth:
-1.4K
Growth Rate:
-126.49%
Total runs:
1.1K
Run Growth:
-1.2K
Growth Rate:
-110.15%
Total runs:
916
Run Growth:
-1.2K
Growth Rate:
-133.30%
Total runs:
824
Run Growth:
329
Growth Rate:
39.93%
Total runs:
783
Run Growth:
783
Growth Rate:
100.00%
Total runs:
212
Run Growth:
-117
Growth Rate:
-55.19%
Total runs:
210
Run Growth:
135
Growth Rate:
64.29%
Total runs:
193
Run Growth:
-130
Growth Rate:
-67.36%
Total runs:
192
Run Growth:
47
Growth Rate:
24.48%
Total runs:
169
Run Growth:
-159
Growth Rate:
-94.08%
Total runs:
136
Run Growth:
4
Growth Rate:
2.86%
Total runs:
59
Run Growth:
-39
Growth Rate:
-66.10%
Total runs:
34
Run Growth:
18
Growth Rate:
52.94%
Total runs:
34
Run Growth:
16
Growth Rate:
47.06%
Total runs:
32
Run Growth:
4
Growth Rate:
12.50%
Total runs:
28
Run Growth:
14
Growth Rate:
50.00%
Total runs:
27
Run Growth:
5
Growth Rate:
18.52%
Total runs:
22
Run Growth:
9
Growth Rate:
40.91%
Total runs:
21
Run Growth:
10
Growth Rate:
47.62%
Total runs:
21
Run Growth:
5
Growth Rate:
23.81%
Total runs:
16
Run Growth:
4
Growth Rate:
25.00%
Total runs:
16
Run Growth:
6
Growth Rate:
37.50%
Total runs:
15
Run Growth:
-6
Growth Rate:
-40.00%
Total runs:
11
Run Growth:
-8
Growth Rate:
-72.73%
Total runs:
8
Run Growth:
2
Growth Rate:
25.00%
Total runs:
8
Run Growth:
-18
Growth Rate:
-225.00%
Total runs:
8
Run Growth:
0
Growth Rate:
0.00%
Total runs:
7
Run Growth:
-2
Growth Rate:
-28.57%
Total runs:
7
Run Growth:
-4
Growth Rate:
-57.14%
Total runs:
4
Run Growth:
-3
Growth Rate:
-75.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%