cstr / voxtral-mini-3b-2507-GGUF

Voxtral-Mini-3B-2507 — GGUF

GGUF / ggml conversions of mistralai/Voxtral-Mini-3B-2507 for use with the voxtral-main CLI from CrispStrobe/CrispASR .

Voxtral Mini is Mistral's 3B-parameter speech-LLM — an enhancement of Ministral 3B with state-of-the-art audio input capabilities while retaining best-in-class text performance. It excels at speech transcription, translation, and audio understanding.

• 8 languages (English, French, German, Spanish, Italian, Portuguese, Dutch, Hindi) with automatic language detection
• Built-in audio Q&A and summarization — ask questions about audio content directly
• Function calling from voice — trigger backend functions based on spoken intents
• Long-form context — up to 30 minutes of audio for transcription, 40 minutes for understanding
• Natively multilingual with state-of-the-art WER across the world's most widely used languages
• Highly capable at text — retains the text understanding capabilities of its Ministral 3B backbone
• Apache-2.0 licence

Files

Both quantisations produce the correct transcript on samples/jfk.wav :

And so, my fellow Americans, ask not what your country can do for you, ask what you can do for your country.

The mel filterbank from WhisperFeatureExtractor and the Tekken tokenizer vocab are baked into the GGUF , so the C++ runtime computes everything natively — no Python/torch/librosa at inference time.

Quick Start

# 1. Build the runtime
git clone https://github.com/CrispStrobe/CrispASR
cd CrispASR
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc) --target voxtral-main

# 2. Download a quantisation
huggingface-cli download cstr/voxtral-mini-3b-2507-GGUF \
    voxtral-mini-3b-2507-q4_k.gguf --local-dir .

# 3. Transcribe
./build/bin/voxtral-main \
    -m voxtral-mini-3b-2507-q4_k.gguf \
    -f audio.wav -t 8 -l en

Audio must be 16 kHz mono 16-bit PCM WAV. Pre-convert with:

ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav

Language selection

Use -l LANG with a two-letter code:

Performance

Measured on samples/jfk.wav (11 seconds), 4-core CPU:

Q4_K gives a 2.2× speedup over F16 while producing identical transcripts. The 3B model is larger than the Qwen3-ASR 0.6B — for fastest CPU inference on short clips, Qwen3-ASR Q4_K (6.6s for 11s audio) is faster; Voxtral's advantage is the richer capabilities (audio understanding, function calling, text Q&A) and superior multilingual WER.

Architecture

Voxtral-Mini-3B is a three-module speech-LLM:

Transcription prompt format

<s> [INST] [BEGIN_AUDIO] <audio_pad>×375 [/INST] lang:en [TRANSCRIBE]

Token IDs: [1, 3, 25, 24×375, 4, 9909, 1058, <lang_id>, 34]

Key differences from standard Whisper

1. Conv1 stride is 1 (Whisper uses stride 2). This means the conv front-end only does 2× temporal reduction (just conv2), not 4×. 3000 mel frames → 1500 encoder frames (vs Whisper's 750).
2. K-proj has no bias in the encoder's self-attention (Whisper quirk preserved from the Whisper-large-v3 weights).
3. The encoder output is not consumed by a Whisper decoder — it's fed through a 4-frame-stack projector into a general-purpose Llama 3 LLM that generates the transcript (or any other text response) autoregressively.

Implementation notes

The C++ runtime is verified against the PyTorch reference (bf16) at every architectural boundary:

The 0.87 min cosine sim on the encoder is from the bf16 reference precision (7-bit mantissa) vs F16 GGUF weights (10-bit) with F32 compute in C++. An F32 reference would give tighter numbers — the end-to-end transcript is the real correctness test and it passes.

Bugs found during the port

1. ggml_conv_1d output layout : returns (OL, OC, N) not (OC, OL) . Bias needs (1, OC, 1) reshape to broadcast over time+batch.
2. Post-conv transpose : ggml_conv_1d puts time on ne[0] , but LayerNorm needs feature dim on ne[0] . Fixed by reshape+transpose to (d, T_enc) .
3. GELU approximation : ggml_gelu (tanh approx) → ggml_gelu_erf (exact) for correctness matching.
4. Tekken vocab blob storage : gguf-py's add_array with Python int lists stores as INT32, corrupting the uint8 byte stream. Fixed by storing as a 1D F32 tensor.

How this was made

1. HF safetensors converted to GGUF F16 by models/convert-voxtral-to-gguf.py . All 765 tensors (762 model + mel_filters + mel_window + Tekken vocab blob) map cleanly.
2. Quantised variants produced by cohere-quantize with the Q4_0 fallback for 1280-wide audio encoder tensors (1280 % 256 ≠ 0 for Q4_K, same situation as Qwen3-ASR).
3. Inference implemented in src/voxtral.{h,cpp} (~1300 LOC): encoder and LLM each run as one ggml graph, with a persistent F16 KV cache (head_dim, max_ctx, n_kv_heads, n_layers) shared between prefill and per-token decode steps. Flash attention ( ggml_flash_attn_ext ) used on both prefill (F16 causal mask) and decode (no mask) paths.

Related

• Original model : mistralai/Voxtral-Mini-3B-2507 (Apache-2.0)
• C++ runtime : CrispStrobe/CrispASR
• Research paper : arxiv.org/abs/2507.13264
• Sister releases in the same family:
  • cstr/qwen3-asr-0.6b-GGUF — Qwen3-ASR 0.6B (faster, 30 languages + Chinese dialects)
  • cstr/parakeet-tdt-0.6b-v3-GGUF — Parakeet TDT 600M (free word timestamps)
  • cstr/canary-1b-v2-GGUF — Canary 978M (speech translation)
  • cstr/cohere-transcribe-03-2026-GGUF — Cohere Transcribe 2B (lowest English WER)

License

Apache-2.0, inherited from the base model.