aufklarer / VoiceChat-11B-Perception-MLX-int5

VoiceChat 11B Perception — MLX 5-bit (output heads 8-bit)

The speech encoder and language backbone of NVIDIA's NemotronLabs VoiceChat 11B, converted to MLX for Apple Silicon.

What this is, and what it is not. This bundle contains the parts that understand speech: the streaming FastConformer encoder, the bridge into the language model, the RNNT transcript head, and the full 9.5 B-parameter Nemotron-H backbone. It does not contain the speech synthesis half — the TTS decoder, the audio codec, and the full-duplex loop are not included, so this cannot hold a spoken conversation on its own. It is a speech understanding front-end plus a language model, not a voice agent.

Model

Files

Performance

Measured against the fp16 bundle these were quantized from, teacher-forced over an identical corpus. Agreement is how often the quantized model picks the same next token as fp16; KL is the divergence of the full next-token distribution (lower is better for both).

This variant: 92.55% agreement, KL 0.01213.

8-bit reproduces the fp16 model's greedy output token for token. 5-bit diverges on roughly one token in thirteen — acceptable for conversational text, but worth measuring on your own task before relying on it for structured output such as tool-call arguments, where a single divergent token invalidates the result.

Perplexity is deliberately not quoted: on a short corpus, quantization noise can lower it without the model being better, so agreement and KL are the honest measures here.

Usage

import mlx.core as mx
from voicechat_mlx import load_llm, load_perception

# Speech understanding: log-mel -> language-model embedding space
encode = load_perception("encoder")
embeddings, lengths = encode(log_mel)     # (B, T, 128) -> (B, T/8, 4480)

# Language backbone
model, tokenizer = load_llm("llm")
ids = mx.array(tokenizer.encode("The capital of Norway is"))
logits = model(ids[None])
print(tokenizer.decode([int(mx.argmax(logits[0, -1]))]))

pip install mlx mlx-lm parakeet-mlx huggingface_hub
hf download aufklarer/VoiceChat-11B-Perception-MLX-int5 --local-dir ./voicechat
python -c "from voicechat_mlx import load_llm; m, t = load_llm('./voicechat/llm'); print('ok')"

The encoder is a NeMo streaming FastConformer and differs from a stock Conformer in three ways that all fail silently if ignored — no biases on the feed-forward, attention and convolution linears; a LayerNorm in place of the convolution BatchNorm; and causal subsampling that keeps 17 frequency bins rather than 16. voicechat_mlx.py handles all three, so use it rather than constructing the module tree yourself.

Source

Converted from nvidia/NVIDIA-NemotronLabs-VoiceChat-11B . The language backbone derives from nvidia/NVIDIA-Nemotron-Nano-9B-v2 , which also supplies the tokenizer. Licensed under OpenMDW 1.1.

Links

• speech-swift — Apple SDK
• Docs — install and CLI docs
• soniqo.audio — website
• blog — blog