hotchpotch / bekko-embedding-v1-a8m-GGUF

bekko-embedding-v1-a8m-GGUF

GGUF release of hotchpotch/bekko-embedding-v1-a8m , an ultra-compact multilingual text embedding model with 7.7M active parameters.

llama.cpp

The GGUF contains a compatible Gemma2 SentencePiece vocabulary and runs on the standard llama.cpp runtime with ModernBERT embedding support:

llama-server \
  -hf hotchpotch/bekko-embedding-v1-a8m-GGUF:Q8_0 \
  --embedding --pooling mean --embd-normalize 2 --ctx-size 8192

curl http://localhost:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"bekko","input":["What is the tallest mountain in Japan?","Mount Fuji is the tallest mountain in Japan."]}'

If the :Q8_0 selector is unavailable in your llama.cpp version, download the GGUF file and pass it with --model :

hf download \
  hotchpotch/bekko-embedding-v1-a8m-GGUF \
  bekko-embedding-v1-a8m-Q8_0.gguf \
  --local-dir .

llama-server \
  --model ./bekko-embedding-v1-a8m-Q8_0.gguf \
  --embedding --pooling mean --embd-normalize 2 --ctx-size 8192

Ollama

Pull the ready-to-use model from the Ollama registry:

ollama pull hotchpotch/bekko-embedding-v1-a8m
curl http://localhost:11434/api/embed \
  -d '{"model":"hotchpotch/bekko-embedding-v1-a8m","input":"What is the tallest mountain in Japan?"}'

The registry artifact has been verified as an embedding model with 384 dimensional, L2-normalized output.

Model contract

• Architecture: ModernBERT encoder
• Context length: 8192 tokens
• Embedding dimension: 384
• Matryoshka dimensions: 256, 128, 64
• Pooling: mean
• Similarity: cosine or dot product after L2 normalization
• Query/document prefixes: none
• Languages: 100+

Use the same unprefixed encoding function for queries and documents.

Quantization quality

Q8_0 was compared with the revision-pinned SentenceTransformers source using the same unprefixed queries and documents:

The mean cosine similarity to the HF embeddings is 0.999695 over five ordinary multilingual samples. Small positive retrieval deltas should be treated as ranking noise, not as an improvement caused by quantization.

Faster CPU inference

GGUF is intended for portable llama.cpp and Ollama deployment. On a fast x86 CPU, the optimized OpenVINO export is generally faster than llama.cpp GGUF and should be preferred when maximum throughput is the priority. It is available in the hotchpotch/bekko-embedding-v1-a8m source repository when maximum throughput is the priority. The source model also provides the original SentenceTransformers model and its full model card.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "hotchpotch/bekko-embedding-v1-a8m",
    backend="openvino",
)
embeddings = model.encode(["What is the tallest mountain in Japan?"], normalize_embeddings=True)

Why only Q8_0?

Q8_0 is the best portable trade-off found in our BF16/Q8_0/Q6_K/Q5_K_M/Q4_K_M/IQ4_XS comparison:

• Lower-bit K-quants provide almost no size reduction because the 384-wide tensors fall back to Q5_0 or Q8_0. Q4_K_M is only 1.9% smaller than Q8_0.
• Lower-bit embeddings have larger cosine drift. IQ4_XS loses 0.0195 mean NDCG@10 on the representative NanoCoIR tasks.
• Q8_0 is the fastest quality-preserving quantization on Raspberry Pi 5. Q4_K_M and Q5_K_M reach only 41% and 39% of Q8_0 throughput respectively.
• Q6_K has the same size, embeddings, benchmark scores, and effectively the same fallback tensor types as Q8_0, so publishing both would be redundant.

Positive benchmark deltas in this small evaluation are ranking noise rather than evidence that lower numerical precision improves the model.

Files and reproducibility

This repository is generated from an immutable source revision. See conversion.json for the exact revision, llama.cpp revision, SHA-256 hash, and release-critical config validation.

For benchmark results, training details, license, and limitations, refer to the source model card .