multimodalart / VibeVoice-ASR-BitNet-ONNX

VibeVoice-ASR-BitNet · ONNX (WebGPU + WASM)

ONNX export of microsoft/VibeVoice-ASR-BitNet for 🤗 Transformers.js / onnxruntime-web — multilingual speech recognition with a ternary (1.58-bit) BitNet Qwen2.5-1.5B decoder, running fully in the browser.

Live demo: multimodalart/vibevoice-asr-bitnet-web

Layout

The repo follows the transformers.js audio-text-to-text (ultravox) layout, so the stock UltravoxModel class drives it with no custom modeling code:

The prompt places one <|speech_pad|> (id 151648) per audio frame ( frames = ceil(samples/3200) ); transformers.js merges audio_features into those positions automatically ( audio_token_id in config.json ).

Quantization (mirrors the GGUF scheme)

The released safetensors are BitNet QAT master weights — they only produce sensible output after ternarization, which is applied exactly before export.

Validation (onnxruntime CPU, greedy, vs ternarized PyTorch reference)

fp32 ONNX: 4/4 transcripts byte-identical. q4 / q4f16 / q2: 3/4 byte-identical; the FLEURS-fr clip differs by two words ("géographe"→"géologue", "des"→"les"). For comparison, the official ggml engine (VibeASR.cpp, I8_S+I2_S) transcribes that same clip as "aux gens ouverts fiscales, mais assètent sur les toits de douane" — these ONNX builds are strictly closer to the fp reference than the original GGUF engine on every tested clip. Full transcripts in conversion_report.json .

Files / sizes

• WebGPU bundle ( q4f16 ): encoder 696 MB + embeddings 234 MB + decoder 869 MB ≈ 1.8 GB
• WASM bundle ( q4 ): ≈ 1.98 GB
• 2-bit decoder ( decoder_model_merged_bnb4.onnx , 726 MB): true I2_S-equivalent; current onnxruntime CPU 2-bit kernels are much slower than 4-bit — published for experimentation.

Usage (transformers.js v4)

import { AutoTokenizer, UltravoxModel, Tensor, TextStreamer } from "@huggingface/transformers";

const model_id = "multimodalart/VibeVoice-ASR-BitNet-ONNX";
const tokenizer = await AutoTokenizer.from_pretrained(model_id);
const model = await UltravoxModel.from_pretrained(model_id, {
  device: "webgpu",            // or "wasm"
  dtype: { audio_encoder: "q4f16", embed_tokens: "q4f16", decoder_model_merged: "q4f16" }, // or all "q4"
});

// audio: Float32Array, 24 kHz mono, RMS-normalized to -25 dBFS, zero-padded to length % 3200 == 0
const frames = audio.length / 3200;
const prompt =
  `<|im_start|>system\nYou are a helpful assistant that transcribes audio input into text output in JSON format.<|im_end|>\n` +
  `<|im_start|>user\n<|speech_start|>${"<|speech_pad|>".repeat(frames)}<|speech_end|>\n` +
  `This is a ${duration} seconds audio, please transcribe it.<|im_end|>\n`;
// build ids via tokenizer (or splice numeric ids 151644/151645/151646/151647/151648 directly)

const out = await model.generate({
  ...tokenizer(prompt, { add_special_tokens: false }),
  audio_values: new Tensor("float32", audio, [1, audio.length]),
  max_new_tokens: 512,
  streamer: new TextStreamer(tokenizer, { skip_prompt: true, skip_special_tokens: true }),
});
// The model emits "<|im_start|>assistant\n" first — strip it from the decoded text.

Provenance

Converted on HF Jobs with an open pipeline: official microsoft/VibeVoice modeling code as reference, exact convert_lm_to_gguf.py ternarization semantics, custom static-causal-padding ONNX export of the tokenizer encoders, structural exact-ternary MatMulNBits packing, transcript-level validation against golden references ( VibeASR.cpp prompt format).