cstr / bge-m3-GGUF

bge-m3 GGUF

GGUF format of BAAI/bge-m3 for use with CrispEmbed .

BGE-M3 is special: a single model produces three retrieval signals simultaneously -- dense, sparse (SPLADE-style lexical weights), and ColBERT-style multi-vector -- over 100+ languages and up to 8192 tokens. CrispEmbed exposes all three heads through one C API; see test_bgem3.py for the validation harness.

Files

Recommended: q8_0 -- effectively lossless for all three heads (see quality below).

Quality vs FlagEmbedding (FP32 reference)

Measured on three test sentences with FlagEmbedding 's BGEM3FlagModel :

Sparse IoU is capped below 1.0 because FlagEmbedding post-filters special tokens ( <s> , </s> ) from the lexical-weight map while CrispEmbed returns them raw. The filter is trivial on the consumer side.

Quick Start

CrispEmbed (CLI)

./crispembed -m bge-m3 "Paris is the capital of France."

# Server (OpenAI/Ollama/native dialects)
./crispembed-server -m bge-m3 --port 8080

CrispEmbed (Python)

from crispembed import CrispEmbed

model = CrispEmbed("bge-m3-q8_0.gguf")

# Dense
dense = model.encode("Paris is the capital of France.")           # [1024]

# Sparse (SPLADE-style lexical weights)
sparse = model.encode_sparse("Paris is the capital of France.")   # [(vocab_id, weight), ...]

# ColBERT / multi-vector (per-token)
colbert = model.encode_multivec("Paris is the capital of France.")  # [n_tokens, 1024]

CrispEmbed (Rust)

use crispembed::CrispEmbed;

let mut m = CrispEmbed::new("bge-m3-q8_0.gguf", 4)?;
let dense   = m.encode("Paris is the capital of France.");
let sparse  = m.encode_sparse("Paris is the capital of France.");
let colbert = m.encode_multivec("Paris is the capital of France.");

Model Details

Server API

CrispEmbed server supports four API dialects:

• POST /embed -- native: {"texts": ["..."]}
• POST /v1/embeddings -- OpenAI-compatible
• POST /api/embed -- Ollama-compatible (batch)
• POST /api/embeddings -- Ollama-compatible (legacy single)

Sparse and ColBERT heads are exposed only through the native C API for now.

Conversion

Produced with the CrispEmbed converter :

python models/convert-bert-to-gguf.py \
    --model BAAI/bge-m3 \
    --output bge-m3.gguf \
    --crisp

./build/crispembed-quantize bge-m3.gguf bge-m3-q8_0.gguf q8_0
./build/crispembed-quantize bge-m3.gguf bge-m3-q4_k.gguf q4_k

The converter downloads sparse_linear.pt / colbert_linear.pt from the HF hub alongside the main XLM-R backbone and stores all metadata CrispEmbed needs for three-headed inference.

Citation

@misc{bge-m3,
  title={BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation},
  author={Jianlv Chen and Shitao Xiao and Peitian Zhang and Kun Luo and Defu Lian and Zheng Liu},
  year={2024},
  eprint={2402.03216},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

Credits

• Original model: BAAI/bge-m3 (MIT, FlagOpen)
• Inference engine: CrispEmbed (MIT, ggml-based)