cstr / vibevoice-asr-GGUF

VibeVoice-ASR — GGUF

GGUF / ggml conversions of microsoft/VibeVoice-ASR for use with the crispasr CLI from CrispStrobe/CrispASR .

VibeVoice-ASR is Microsoft's 7B-parameter speech-LLM capable of transcribing up to 60 minutes of audio in a single pass . Unlike most ASR models it outputs structured JSON containing who (speaker diarization), when (timestamps), and what (content) simultaneously — with support for customised hotwords and 50+ languages .

• 60-minute long-form audio in a single forward pass (streaming segmentation for >60 s)
• Built-in speaker diarization — Speaker IDs in every output segment
• Word-level timestamps — Start time / End time per segment
• Hotword / context injection — pass terms or metadata via --context
• 50+ languages with automatic language detection
• MIT licence

Files

Quick Start

# 1. Build CrispASR
git clone https://github.com/CrispStrobe/CrispASR
cd CrispASR
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON   # macOS
cmake --build build -j$(nproc)

# 2. Download the quantised GGUF
huggingface-cli download cstr/vibevoice-asr-GGUF \
    vibevoice-asr-q4_k.gguf --local-dir .

# 3. Transcribe
./build/bin/crispasr --model vibevoice-asr-q4_k.gguf \
    --file audio.wav --backend vibevoice

Audio must be 24 kHz mono. Pre-convert with:

ffmpeg -i input.mp3 -ar 24000 -ac 1 -c:a pcm_s16le output.wav

With hotwords / context

./build/bin/crispasr --model vibevoice-asr-q4_k.gguf \
    --file meeting.wav --backend vibevoice \
    --context "ACME Corp, John Smith, Q3 earnings"

Output Format

The model outputs structured JSON per segment:

[
  {
    "start_time": "0:00:01.234",
    "end_time":   "0:00:04.567",
    "speaker_id": "Speaker_1",
    "text":       "And so, my fellow Americans …"
  },
  …
]

This differs from most ASR models — the LLM generates the JSON itself; no separate diarization step is needed.

Architecture

VibeVoice-ASR is a four-module speech-LLM:

Prompt format

<|im_start|>system
You are a helpful assistant that transcribes audio input into text output in JSON format.
<|im_end|>
<|im_start|>user
<|speech_start|><|speech_pad|>×N<|speech_end|>
This is a X.XX seconds audio, please transcribe it with these keys: Start time, End time, Speaker ID, Content
<|im_end|>
<|im_start|>assistant

Where N = ceil(samples / 3200) .

Audio preprocessing

1. Resample to 24 kHz mono
2. Normalise to −25 dBFS RMS: audio *= 10^(−25/20) / (rms + 1e-6) , then clip if |audio| > 1

What's baked into the GGUF

• All acoustic encoder weights (F16 for conv weights except depthwise which stays F32)
• Both SpeechConnectors
• Full Qwen2-7B LLM weights
• Qwen2.5 tokenizer vocabulary (embedded as tokenizer.ggml.tokens )
• Architecture hyperparameters as GGUF metadata keys

Implementation notes

The C++ runtime validates against the PyTorch reference at every pipeline boundary using tools/dump_reference.py --backend vibevoice :

Note on acoustic sampling : the Python model uses std_dist_type='gaussian' (adds per-batch noise during training/inference for robustness). The C++ runtime uses the VAE mean directly (deterministic), which is equivalent to dist_type='none' and gives reproducible, comparable output for diff-testing.

How this was made

1. microsoft/VibeVoice-ASR safetensors converted to GGUF F16 by models/convert-vibevoice-to-gguf.py . Tensors loaded in native dtype (BF16) to avoid OOM on the large embedding table; converted to F16/F32 per-tensor at write time.
2. Quantised variants produced by the CrispASR quantize binary (Q4_K_M).
3. Full pipeline implemented in src/vibevoice.{h,cpp} : two encoder graphs + two connector graphs + Qwen2 autoregressive decoder with F16 KV cache.

Related

• Original model : microsoft/VibeVoice-ASR (MIT)
• Reference code : microsoft/VibeVoice
• C++ runtime : CrispStrobe/CrispASR
• Technical report : arxiv.org/pdf/2601.18184
• Sister releases:
  • cstr/qwen3-asr-0.6b-GGUF — Qwen3-ASR 0.6B (fast, 30 languages)
  • cstr/voxtral-mini-3b-2507-GGUF — Voxtral Mini 3B (audio Q&A)
  • cstr/granite-speech-3.3-8b-GGUF — Granite Speech 8B
  • cstr/parakeet-tdt-0.6b-v3-GGUF — Parakeet TDT 600M

License

MIT, inherited from the base model.