mlboydaisuke / bge-reranker-base-ExecuTorch

bge-reranker-base — ExecuTorch

A cross-encoder reranker: a query and one document in, one relevance score out. The second stage of on-device retrieval — an embedding model fetches candidates cheaply, this reads each candidate together with the query and scores it properly.

• Source : BAAI/bge-reranker-base — 278M parameters, 12 XLM-RoBERTa layers, hidden 768, 250k vocabulary
• License : MIT
• Input : input_ids and attention_mask , both [1, 512] int64
• Output : [1, 1] fp32 — the raw logit. sigmoid(x) maps it to 0..1 and does not change the ordering.

Variants

*Mac arm64, single process, median of 10, one query-document pair at 512 tokens — a reference point for relative cost, not a device number. PyTorch eager fp32 on the same machine: 54.7 ms.

Correlation is not reported because it cannot be: the output is a single number, and the correlation of a one-element vector is undefined. The column above is the error in the units the model is used in — logits — over 6 real query-document pairs, and every build listed reproduces eager's ranking order exactly.

What it does, on the shipped fp32 build

Query: "How many people live in Berlin?"

The narrowest gap between adjacent ranks here is 0.0057 logits — the top two both answer the question, so their order is a coin toss and a build that swapped them would not be wrong.

It ranks across languages

The candidate list above includes a Japanese passage that answers the English query. This model puts it first at +10.31; ms-marco-MiniLM-L6, the English-only reranker on this shelf, scores the same passage -10.96 and puts it fifth of 6. That is what the 250k-token vocabulary buys, and it is also why this file is 12 times larger.

Not shipped

• int8 (dynamic) is not shipped : at 856.1 MB it is larger than the fp16 build's 556.4 MB, and its score error is 0.5776 logits. Dynamic int8 quantizes the linear weights and leaves the token embedding table in fp32, while fp16 halves that table too. The table here is 768 MB of a 1112 MB model, and the arithmetic says int8 only comes out smaller when the table is under a third of the weights (371 MB) — measured on ten models on this shelf, the rule called all ten correctly.

Verification

python convert/export_rerank.py bge_reranker_base
python convert/check_rerank.py bge_reranker_base fp32

The check has two halves. One is agreement with the model run in eager, in logits and in ranking order. The other is that the ranking is useful at all: the passage that answers the question has to outscore a passage about the same subject that does not — agreement alone would pass a build that ranked by document length in both arms.

(conversion scripts: executorch-models )