Total runs:
5.5M
Run Growth:
1.2M
Growth Rate:
21.15%
Sponsored by
PoYo.AI
microsoft / VibeVoice-AcousticTokenizer
Model's Last Updated: February 06 2026
feature-extraction
Introduction of VibeVoice-AcousticTokenizer
Model Details of VibeVoice-AcousticTokenizer
VibeVoice Acoustic Tokenizer
VibeVoice is a novel framework designed for generating expressive, long-form, multi-speaker conversational audio, such as podcasts, from text. It addresses significant challenges in traditional Text-to-Speech (TTS) systems, particularly in scalability, speaker consistency, and natural turn-taking.
A core innovation of VibeVoice is its use of continuous speech tokenizers (Acoustic and Semantic) operating at an ultra-low frame rate of 7.5 Hz. These tokenizers efficiently preserve audio fidelity while significantly boosting computational efficiency for processing long sequences. VibeVoice employs a next-token diffusion framework, leveraging a Large Language Model (LLM) to understand textual context and dialogue flow, and a diffusion head to generate high-fidelity acoustic details.
The speech tokenizer is a key component for both VibeVoice TTS and ASR .
➡️ Technical Report: VibeVoice Technical Report
➡️ Project Page: microsoft/VibeVoice
Models
| Model | Context Length | Length (min) | Weight |
|---|---|---|---|
| VibeVoice-Realtime-0.5B | 8K | ~10 min | HF link |
| VibeVoice-1.5B | 64K | ~90 min | HF link |
| VibeVoice-ASR | 64K | ~60 min | HF link |
| VibeVoice-AcousticTokenizer | - | - | This model |
Usage
Setup
Until the VibeVoice acoustic tokenizer is part of an official Transformers release, it can be used by installing from the source code:
pip install git+https://github.com/huggingface/transformers.git
Example
Encoding and decoding
import torch
from scipy.io import wavfile
from transformers import AutoFeatureExtractor, VibeVoiceAcousticTokenizerModel
from transformers.audio_utils import load_audio_librosa
model_id = "microsoft/VibeVoice-AcousticTokenizer"
# load model
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
model = VibeVoiceAcousticTokenizerModel.from_pretrained(model_id, device_map="auto")
print("Model loaded on device:", model.device)
print("Model dtype:", model.dtype)
# load audio
audio = load_audio_librosa(
"https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/voices/en-Alice_woman.wav",
sampling_rate=feature_extractor.sampling_rate,
)
# preprocess audio
inputs = feature_extractor(
audio,
sampling_rate=feature_extractor.sampling_rate,
pad_to_multiple_of=3200,
).to(model.device, model.dtype)
print("Input audio shape:", inputs.input_values.shape)
# Input audio shape: torch.Size([1, 1, 224000])
with torch.no_grad():
# set VAE sampling to False for deterministic output
encoded_outputs = model.encode(inputs.input_values, sample=False)
print("Latent shape:", encoded_outputs.latents.shape)
# Latent shape: torch.Size([1, 70, 64])
decoded_outputs = model.decode(**encoded_outputs)
print("Reconstructed audio shape:", decoded_outputs.audio.shape)
# Reconstructed audio shape: torch.Size([1, 1, 224000])
# Save audio
output_fp = "vibevoice_acoustic_tokenizer_reconstructed.wav"
wavfile.write(output_fp, feature_extractor.sampling_rate, decoded_outputs.audio.squeeze().float().cpu().numpy())
print(f"Reconstructed audio saved to : {output_fp}")
Original audio
Encoded/decoded audio
Streaming
For streaming ASR or TTS, where cached states need to be tracked, the
use_cache
parameter can be used when encoding or decoding audio:
import torch
from scipy.io import wavfile
from transformers import AutoFeatureExtractor, VibeVoiceAcousticTokenizerModel
from transformers.audio_utils import load_audio_librosa
model_id = "microsoft/VibeVoice-AcousticTokenizer"
# load model
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
model = VibeVoiceAcousticTokenizerModel.from_pretrained(model_id, device_map="auto")
print("Model loaded on device:", model.device)
print("Model dtype:", model.dtype)
# load audio
audio = load_audio_librosa(
"https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/voices/en-Alice_woman.wav",
sampling_rate=feature_extractor.sampling_rate,
)
# preprocess audio
inputs = feature_extractor(
audio,
sampling_rate=feature_extractor.sampling_rate,
pad_to_multiple_of=3200,
).to(model.device, model.dtype)
print("Input audio shape:", inputs.input_values.shape)
# Input audio shape: torch.Size([1, 1, 224000])
# chache will be initialized after a first pass
encoder_cache = None
decoder_cache = None
with torch.no_grad():
# set VAE sampling to False for deterministic output
encoded_outputs = model.encode(inputs.input_values, sample=False, padding_cache=encoder_cache, use_cache=True)
print("Latent shape:", encoded_outputs.latents.shape)
# Latent shape: torch.Size([1, 70, 64])
decoded_outputs = model.decode(encoded_outputs.latents, padding_cache=decoder_cache, use_cache=True)
print("Reconstructed audio shape:", decoded_outputs.audio.shape)
# Reconstructed audio shape: torch.Size([1, 1, 224000])
# `padding_cache` can be extracted from the outputs for subsequent passes
encoder_cache = encoded_outputs.padding_cache
print("Number of cached encoder layers:", len(encoder_cache.per_layer_in_channels))
# Number of cached encoder layers: 34
decoder_cache = decoded_outputs.padding_cache
print("Number of cached decoder layers:", len(decoder_cache.per_layer_in_channels))
# Number of cached decoder layers: 34
# Save audio
output_fp = "vibevoice_acoustic_tokenizer_reconstructed.wav"
wavfile.write(output_fp, feature_extractor.sampling_rate, decoded_outputs.audio.squeeze().float().cpu().numpy())
print(f"Reconstructed audio saved to : {output_fp}")
Runs of microsoft VibeVoice-AcousticTokenizer on huggingface.co
529
Total runs
2
24-hour runs
13
3-day runs
16
7-day runs
-423
30-day runs
More Information About VibeVoice-AcousticTokenizer huggingface.co Model
Url of VibeVoice-AcousticTokenizer
VibeVoice-AcousticTokenizer huggingface.co Url
Provider of VibeVoice-AcousticTokenizer huggingface.co
Other API from microsoft
Updated:April 06 2023
Total runs:
3.1M
Run Growth:
319.4K
Growth Rate:
10.45%
Total runs:
2.8M
Run Growth:
-132.1K
Growth Rate:
-4.75%
Total runs:
2.1M
Run Growth:
-585.3K
Growth Rate:
-27.85%
Total runs:
1.3M
Run Growth:
37.6K
Growth Rate:
2.83%
Total runs:
1.2M
Run Growth:
-63.5K
Growth Rate:
-5.41%
Total runs:
995.8K
Run Growth:
-147.5K
Growth Rate:
-14.81%
Total runs:
839.0K
Run Growth:
-738.7K
Growth Rate:
-88.04%
Total runs:
838.5K
Run Growth:
-286.4K
Growth Rate:
-34.15%
Total runs:
808.5K
Run Growth:
767
Growth Rate:
0.09%
Total runs:
767.5K
Run Growth:
-1.4K
Growth Rate:
-0.18%
Total runs:
733.9K
Run Growth:
34.3K
Growth Rate:
4.67%
Total runs:
681.1K
Run Growth:
208.0K
Growth Rate:
30.53%
Total runs:
643.3K
Run Growth:
50.5K
Growth Rate:
7.85%
Total runs:
609.9K
Run Growth:
241.2K
Growth Rate:
39.54%
Total runs:
595.8K
Run Growth:
-131.1K
Growth Rate:
-22.00%
Total runs:
513.7K
Run Growth:
-134.7K
Growth Rate:
-26.22%
Total runs:
511.8K
Run Growth:
-531.5K
Growth Rate:
-103.84%
Total runs:
432.9K
Run Growth:
176.8K
Growth Rate:
40.84%
Total runs:
424.7K
Run Growth:
-194.1K
Growth Rate:
-45.01%
Total runs:
413.2K
Run Growth:
26.9K
Growth Rate:
6.51%
Total runs:
380.5K
Run Growth:
362.4K
Growth Rate:
95.24%
Total runs:
377.9K
Run Growth:
-70.9K
Growth Rate:
-18.75%
Total runs:
375.6K
Run Growth:
-1.1M
Growth Rate:
-285.61%
Total runs:
361.9K
Run Growth:
84.8K
Growth Rate:
23.42%
Total runs:
333.9K
Run Growth:
-285.3K
Growth Rate:
-85.45%
Total runs:
327.9K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
307.3K
Run Growth:
56.8K
Growth Rate:
18.49%
Total runs:
293.2K
Run Growth:
64.8K
Growth Rate:
22.11%
Total runs:
289.1K
Run Growth:
-22.3K
Growth Rate:
-7.70%
Total runs:
287.0K
Run Growth:
40.9K
Growth Rate:
14.24%
Total runs:
279.2K
Run Growth:
34.6K
Growth Rate:
12.40%
Total runs:
274.9K
Run Growth:
-434.4K
Growth Rate:
-158.04%
Total runs:
265.7K
Run Growth:
-97.0K
Growth Rate:
-36.50%
Total runs:
255.7K
Run Growth:
-259.8K
Growth Rate:
-101.60%
Total runs:
255.3K
Run Growth:
-5.3K
Growth Rate:
-2.08%
Total runs:
254.5K
Run Growth:
142.7K
Growth Rate:
56.04%
Total runs:
237.5K
Run Growth:
35.3K
Growth Rate:
14.85%
Total runs:
230.6K
Run Growth:
-26.8K
Growth Rate:
-11.64%
Total runs:
225.9K
Run Growth:
-226.3K
Growth Rate:
-100.18%
Total runs:
217.5K
Run Growth:
48.6K
Growth Rate:
22.34%
Total runs:
211.9K
Run Growth:
-19.5K
Growth Rate:
-9.12%
Total runs:
187.1K
Run Growth:
15.3K
Growth Rate:
8.17%
Total runs:
185.5K
Run Growth:
-313.4K
Growth Rate:
-168.88%
Total runs:
180.1K
Run Growth:
24.8K
Growth Rate:
13.77%
Total runs:
169.2K
Run Growth:
-25.7K
Growth Rate:
-15.18%
Total runs:
168.6K
Run Growth:
10.2K
Growth Rate:
6.07%
Total runs:
159.2K
Run Growth:
-46
Growth Rate:
-0.03%
Total runs:
152.2K
Run Growth:
-237.9K
Growth Rate:
-156.31%
Total runs:
139.6K
Run Growth:
68.9K
Growth Rate:
49.38%
Total runs:
138.5K
Run Growth:
19.5K
Growth Rate:
14.09%
Total runs:
136.2K
Run Growth:
-216.6K
Growth Rate:
-159.09%
Total runs:
135.6K
Run Growth:
51.3K
Growth Rate:
37.87%
Total runs:
123.7K
Run Growth:
-51.4K
Growth Rate:
-41.55%
Total runs:
123.4K
Run Growth:
13.0K
Growth Rate:
10.56%
Total runs:
122.1K
Run Growth:
-10.1K
Growth Rate:
-8.28%
Total runs:
117.8K
Run Growth:
-147.4K
Growth Rate:
-125.12%
Total runs:
117.6K
Run Growth:
-1.2K
Growth Rate:
-1.03%
Total runs:
109.2K
Run Growth:
3.7K
Growth Rate:
3.37%
Total runs:
105.4K
Run Growth:
-2.5K
Growth Rate:
-2.42%
Total runs:
100.5K
Run Growth:
54.7K
Growth Rate:
54.46%
Total runs:
99.4K
Run Growth:
-20.8K
Growth Rate:
-20.95%
Total runs:
92.4K
Run Growth:
-15.5K
Growth Rate:
-16.80%
Total runs:
88.9K
Run Growth:
-1.8K
Growth Rate:
-2.01%
Total runs:
80.8K
Run Growth:
14.3K
Growth Rate:
17.65%
Total runs:
78.2K
Run Growth:
2.5K
Growth Rate:
3.24%
Total runs:
70.7K
Run Growth:
8.3K
Growth Rate:
11.73%
Total runs:
69.7K
Run Growth:
-13.1K
Growth Rate:
-18.74%
Total runs:
68.7K
Run Growth:
-18.2K
Growth Rate:
-26.47%
Total runs:
62.8K
Run Growth:
-94.3K
Growth Rate:
-150.13%
Total runs:
62.4K
Run Growth:
20.0K
Growth Rate:
32.09%
Total runs:
58.1K
Run Growth:
-9.3K
Growth Rate:
-15.95%
Total runs:
56.3K
Run Growth:
40.7K
Growth Rate:
72.26%
Total runs:
55.7K
Run Growth:
-1.8K
Growth Rate:
-3.23%
Total runs:
54.2K
Run Growth:
-61.9K
Growth Rate:
-114.24%
Total runs:
49.7K
Run Growth:
5.0K
Growth Rate:
10.06%
Total runs:
47.7K
Run Growth:
-590
Growth Rate:
-1.24%
Total runs:
46.4K
Run Growth:
15.4K
Growth Rate:
33.16%
Total runs:
45.8K
Run Growth:
881
Growth Rate:
1.92%
Total runs:
43.7K
Run Growth:
21.0K
Growth Rate:
48.21%
Total runs:
42.8K
Run Growth:
21.5K
Growth Rate:
50.09%
Total runs:
41.9K
Run Growth:
18.0K
Growth Rate:
43.06%
Total runs:
41.7K
Run Growth:
-8.2K
Growth Rate:
-19.66%
Total runs:
37.3K
Run Growth:
15.5K
Growth Rate:
41.46%
Total runs:
35.1K
Run Growth:
-9.1K
Growth Rate:
-25.79%
Total runs:
35.0K
Run Growth:
5.2K
Growth Rate:
14.85%
Total runs:
32.9K
Run Growth:
1.7K
Growth Rate:
5.15%
Total runs:
32.1K
Run Growth:
30.7K
Growth Rate:
95.72%
Total runs:
30.7K
Run Growth:
-3.3K
Growth Rate:
-10.74%
Total runs:
30.7K
Run Growth:
-135.7K
Growth Rate:
-442.61%
Total runs:
28.8K
Run Growth:
-13.8K
Growth Rate:
-48.00%
Total runs:
28.1K
Run Growth:
15.7K
Growth Rate:
55.87%
Total runs:
26.8K
Run Growth:
5.1K
Growth Rate:
19.10%