hotchpotch / bekko-embedding-v1-a8m

Scores are ×100. Retrieval is task-macro nDCG@10, and Mean is the mean across all 131 tasks. Competitor values use the official 2026-06-28 snapshot. Bekko was evaluated over the same task set and aggregation rules.

What each column means:

• Overall — HAKARI-Bench Overall, the micro-average across all the sets below
• MNanoBEIR — multilingual NanoBEIR, general-purpose retrieval
• NanoMMTEB-v2 — Nano subset of MMTEB v2 (massive multilingual retrieval)
• NanoRTEB — multilingual retrieval benchmark
• NanoLongEmbed — long-document retrieval
• NanoCoIR — code retrieval

Document throughput uses Natural Questions text, batch size 64 and max length 512 for CPU/MPS. CUDA uses NQ 100k, fp16 , and Flash Attention 2. All throughput values in the table are docs/s.

Abbreviations: mE5 = multilingual-e5, granite-97m/311m-r2 = Granite Embedding Multilingual R2, EmbGemma = EmbeddingGemma. x86 = Ryzen 9 7950X + OpenVINO, Pi 5 = Raspberry Pi 5 + OpenVINO, M4 = Apple M4 Max + MPS, RTX = RTX 5090 + CUDA/Flash Attention 2. AP means active parameters.

Throughput depends on input lengths, batch size, runtime, and hardware. OpenVINO is recommended for CPU, MPS for Apple Silicon, and Flash Attention 2 for supported NVIDIA GPUs.

SDPA works everywhere and is the safe default. If your GPU supports Flash Attention 2, it's worth enabling: on our RTX 5090 it was about 18% faster than SDPA for a8m (24% for a25m).

import torch
from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer(
    "hotchpotch/bekko-embedding-v1-a8m",
    device="cuda",
    model_kwargs={
        "attn_implementation": "flash_attention_2",
        "dtype": torch.float16,
    },
)

query = "What are the characteristics of sushi?"
docs = [
    "A warm noodle soup served in broth with sliced toppings.",
    "天ぷらは魚や野菜に衣をつけて揚げた料理です。",  # "Tempura is battered, deep-fried fish and vegetables."
    "Une fine crepe garnie de sucre, de beurre ou de fruits.",
    "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
]

scores = util.cos_sim(
    model.encode(query, normalize_embeddings=True),
    model.encode(docs, normalize_embeddings=True),
)[0]
print(scores)

If Flash Attention 2 is unavailable, use model_kwargs={"attn_implementation": "sdpa"} .

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer(
    "hotchpotch/bekko-embedding-v1-a8m",
    device="mps",
    model_kwargs={"attn_implementation": "sdpa"},
)

query = "What are the characteristics of sushi?"
docs = [
    "A warm noodle soup served in broth with sliced toppings.",
    "天ぷらは魚や野菜に衣をつけて揚げた料理です。",  # "Tempura is battered, deep-fried fish and vegetables."
    "Une fine crepe garnie de sucre, de beurre ou de fruits.",
    "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
]

scores = util.cos_sim(
    model.encode(query, normalize_embeddings=True),
    model.encode(docs, normalize_embeddings=True),
)[0]
print(scores)

The default IR ( openvino/openvino_model.xml + .bin ) runs on Intel, AMD, and Arm CPUs, Raspberry Pi included. Only the static token embedding table is stored as int8 — the transformer layers stay fp32 , so quality is essentially unchanged (cosine similarity ≥ 0.9994 to PyTorch in our release checks).

# As of 2026-07-28, Transformers 4.x must be specified so that pip resolves
# a compatible OpenVINO dependency stack.
pip install -U \
  "sentence-transformers[openvino]>=5.0" \
  "transformers>=4.57,<5"

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer(
    "hotchpotch/bekko-embedding-v1-a8m",
    backend="openvino",
    device="cpu",
    model_kwargs={"file_name": "openvino/openvino_model.xml", "device": "CPU"},
)

query = "What are the characteristics of sushi?"
docs = [
    "A warm noodle soup served in broth with sliced toppings.",
    "天ぷらは魚や野菜に衣をつけて揚げた料理です。",  # "Tempura is battered, deep-fried fish and vegetables."
    "Une fine crepe garnie de sucre, de beurre ou de fruits.",
    "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
]

scores = util.cos_sim(
    model.encode(query, normalize_embeddings=True),
    model.encode(docs, normalize_embeddings=True),
)[0]
print(scores)

The default ONNX artifact ( onnx/model.onnx ) keeps the tokenizer and full vocabulary unchanged and stores only the static token embedding table as int8 . Use it with Transformers.js in the browser, or with ONNX Runtime. If plain CPU throughput is what you're after, OpenVINO above is faster. For a complete client-side example, see the bekko-embedding-web Space.

npm install @huggingface/transformers

import { pipeline } from "@huggingface/transformers";

// Browser: use WebGPU when available, otherwise fall back to WASM.
// Node.js: replace this line with `const device = "cpu";`.
const device = navigator.gpu ? "webgpu" : "wasm";

const extractor = await pipeline(
  "feature-extraction",
  "hotchpotch/bekko-embedding-v1-a8m",
  {
    device,
    // Transformers.js maps dtype="fp32" to onnx/model.onnx.
    // In this repo, that file is the compact static-embedding-int8 ONNX model.
    dtype: "fp32",
  },
);

const queryEmbedding = await extractor("What are the characteristics of sushi?", {
  pooling: "mean",
  normalize: true,
});

const documentEmbedding = await extractor(
  "A Japanese dish made with vinegared rice and seafood.",
  { pooling: "mean", normalize: true },
);

console.log(queryEmbedding.tolist()[0].slice(0, 8));
console.log(documentEmbedding.tolist()[0].slice(0, 8));
console.log(queryEmbedding.dims); // [1, 384]

pip install -U "sentence-transformers[onnx]>=5.0"

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer(
    "hotchpotch/bekko-embedding-v1-a8m",
    backend="onnx",
    device="cpu",
    model_kwargs={"file_name": "onnx/model.onnx", "provider": "CPUExecutionProvider"},
)

query = "What are the characteristics of sushi?"
docs = [
    "A warm noodle soup served in broth with sliced toppings.",
    "天ぷらは魚や野菜に衣をつけて揚げた料理です。",  # "Tempura is battered, deep-fried fish and vegetables."
    "Une fine crepe garnie de sucre, de beurre ou de fruits.",
    "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
]
print(util.cos_sim(model.encode(query, normalize_embeddings=True), model.encode(docs, normalize_embeddings=True))[0])

These models are trained with Matryoshka representation learning, so you can shrink the 384-dim embeddings to 256, 128, or 64 dimensions by passing truncate_dim . Smaller dimensions reduce index size and speed up similarity search, at a small cost in retrieval quality (see Truncation and Quantization ).

from sentence_transformers import SentenceTransformer, util

# Full embedding is 384-dim; 256 / 128 / 64 are supported.
model = SentenceTransformer(
    "hotchpotch/bekko-embedding-v1-a8m",
    truncate_dim=256,
    model_kwargs={"attn_implementation": "sdpa"},
)

query = "What are the characteristics of sushi?"
docs = [
    "A warm noodle soup served in broth with sliced toppings.",
    "天ぷらは魚や野菜に衣をつけて揚げた料理です。",  # "Tempura is battered, deep-fried fish and vegetables."
    "Une fine crepe garnie de sucre, de beurre ou de fruits.",
    "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
]
emb = model.encode(query, normalize_embeddings=True)
print("embedding dim:", emb.shape[-1])
print(util.cos_sim(emb, model.encode(docs, normalize_embeddings=True))[0])

Not the same as the default artifact above — this one quantizes the transformer weights too. We keep it for experimentation only: on models this small, qint8 tends to hurt retrieval quality without reliably improving latency.

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer(
    "hotchpotch/bekko-embedding-v1-a8m",
    backend="openvino",
    device="cpu",
    model_kwargs={"file_name": "openvino/openvino_model_qint8_not_recommended.xml", "device": "CPU"},
)

query = "What are the characteristics of sushi?"
docs = [
    "A warm noodle soup served in broth with sliced toppings.",
    "天ぷらは魚や野菜に衣をつけて揚げた料理です。",  # "Tempura is battered, deep-fried fish and vegetables."
    "Une fine crepe garnie de sucre, de beurre ou de fruits.",
    "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
]
print(util.cos_sim(model.encode(query, normalize_embeddings=True), model.encode(docs, normalize_embeddings=True))[0])

Same caveat as OpenVINO qint8 : these files quantize the transformer weights and are platform-specific experiments. On models this small they can noticeably degrade retrieval quality, so measure on your target hardware before adopting them.

pip install -U "sentence-transformers[onnx]>=5.0"

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer(
    "hotchpotch/bekko-embedding-v1-a8m",
    backend="onnx",
    device="cpu",
    model_kwargs={
        "file_name": "onnx/model_qint8_avx512_not_recommended.onnx",
        "provider": "CPUExecutionProvider",
    },
)

query = "What are the characteristics of sushi?"
docs = [
    "A warm noodle soup served in broth with sliced toppings.",
    "天ぷらは魚や野菜に衣をつけて揚げた料理です。",  # "Tempura is battered, deep-fried fish and vegetables."
    "Une fine crepe garnie de sucre, de beurre ou de fruits.",
    "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
]
print(util.cos_sim(model.encode(query, normalize_embeddings=True), model.encode(docs, normalize_embeddings=True))[0])

Text Embeddings Inference (TEI) is Hugging Face's Rust-based serving stack, with official Docker images, dynamic batching, and Prometheus metrics built in.

Before deploying, confirm your TEI version supports this model's encoder architecture, and pick the image tag that matches your target — a CPU image, or a GPU image for your specific architecture. See the TEI image list for current tags.

model=hotchpotch/bekko-embedding-v1-a8m
volume=$PWD/tei-data
# Replace <tag> with the current TEI image for your hardware (CPU, or your GPU arch).
# Add `--gpus all` when using a GPU image.
docker run -p 8080:80 -v "$volume:/data" --pull always \
  ghcr.io/huggingface/text-embeddings-inference:<tag> \
  --model-id "$model"

import requests
import numpy as np

query = "What are the characteristics of sushi?"
docs = [
    "A warm noodle soup served in broth with sliced toppings.",
    "天ぷらは魚や野菜に衣をつけて揚げた料理です。",  # "Tempura is battered, deep-fried fish and vegetables."
    "Une fine crepe garnie de sucre, de beurre ou de fruits.",
    "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
]

q = np.array(requests.post("http://127.0.0.1:8080/embed", json={"inputs": [query]}).json()[0])
d = np.array(requests.post("http://127.0.0.1:8080/embed", json={"inputs": docs}).json())
q = q / np.linalg.norm(q)
d = d / np.linalg.norm(d, axis=1, keepdims=True)
print(d @ q)

Apply mean pooling with pure Transformers.

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model_id = "hotchpotch/bekko-embedding-v1-a8m"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, attn_implementation="sdpa").eval()

def embed(texts):
    batch = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")
    with torch.no_grad():
        out = model(**batch).last_hidden_state
    mask = batch["attention_mask"].unsqueeze(-1)
    pooled = (out * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1)
    return F.normalize(pooled, p=2, dim=1)

query = "What are the characteristics of sushi?"
docs = [
    "A warm noodle soup served in broth with sliced toppings.",
    "天ぷらは魚や野菜に衣をつけて揚げた料理です。",  # "Tempura is battered, deep-fried fish and vegetables."
    "Une fine crepe garnie de sucre, de beurre ou de fruits.",
    "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
]

scores = embed(docs) @ embed([query]).T
print(scores.squeeze(-1))

• Bekko is optimized primarily for multilingual retrieval. Its strongest MMTEB results are Retrieval, Reranking, BitextMining, and STS. It is not intended to be state of the art across every embedding task category.
• Bekko is a bi-encoder embedding model, not a cross-encoder reranker. MMTEB Reranking scores measure bi-encoder similarity scoring. Use a dedicated cross-encoder when maximum reranking accuracy is more important than throughput.
• Support for 100+ languages reflects training-data coverage. Quality varies by language and domain, so evaluate on your target data before deployment.
• HAKARI-Bench is maintained by the model author and should be read alongside the independently maintained MMTEB suite. Bekko's MMTEB results use the same 131-task set and aggregation rules as the referenced snapshot, but await submission through the official leaderboard pipeline.
• Throughput varies with text lengths, batch size, backend, software versions, and hardware. Use the benchmark figures as comparative measurements, not guaranteed production latency.
• Transformer-weight qint8 artifacts are experimental and can lose retrieval quality or behave differently across CPU architectures. The default ONNX/OpenVINO artifacts only compress the static token embedding table and are the recommended deployment files.