mlboydaisuke / multilingual-e5-base-ExecuTorch

multilingual-e5-base — ExecuTorch

Multilingual sentence embeddings: text in any of about a hundred languages in, one 768-dimensional vector out, comparable across languages. For search and retrieval that never leaves the device.

• Source : intfloat/multilingual-e5-base — 278M parameters, 12 XLM-RoBERTa layers, 250k vocabulary, 768-dimensional output
• License : MIT
• Input : input_ids and attention_mask , both [1, 256] int64
• Output : [1, 768] , mean-pooled over the mask and L2-normalised inside the graph

Two parts of the recipe, and only one of them is in the graph

The pooling is in. sentence-transformers keeps it per model, and the six embedding models on this shelf do not agree:

Read off modules.json and 1_Pooling/config.json rather than assumed from the family name — paraphrase-multilingual is the same architecture family with the same pooling, and it does not normalise.

The prefix is not. E5 is trained with "query: " in front of a search query and "passage: " in front of a document, and it expects them at inference:

query:   query: how do I keep data on the phone?
passage: passage: On-device inference keeps the data on the phone.

That is text, so it happens before tokenisation and the .pte never sees it as anything but tokens. Leaving it out does not throw and does not look wrong — it returns a plausible vector that retrieves worse. The conversion repo's checker applies it, so the numbers below are for the recipe as the model intends it.

Verification

Mac arm64, median of 10, one 256-token sequence — a reference point for relative cost, not a device number. Eager fp32 on the same input is 32.9 ms. Cosine is measured against the model run in eager through its own pooling, over eight sentences.

And that the vectors are useful , which agreement alone cannot show. A paraphrase against an unrelated sentence, and then the same test across languages — "機械学習のモデルを端末の上で動かす" against "On-device inference keeps the data on the phone":

same language:   0.848 same meaning  vs  0.669 unrelated
across languages: 0.781 same meaning  vs  0.701 unrelated

The cross-lingual row is what this model is for, and it is the row the English-only models on this shelf fail. Note the scale: E5 puts everything high, so 0.70 for an unrelated pair is normal and the gap is what carries the signal, not the absolute number.

python convert/check_embed.py multilingual_e5_base fp32     # or fp16, int8, coreml

Not shipped

int8 converts and holds — worst cosine 0.996673, and it still separates the pairs — but it comes out at 855.6 MB against fp16's 555.2 MB . Dynamic int8 quantises the linear weights and leaves the token embedding table alone, and with a 250k vocabulary at 768 dimensions that table is 768 MB of the 1110 MB model , 69% of it. Quantising every linear saves 254 MB; fp16 halves the table too. This shelf's rule of thumb holds: int8 beats fp16 only when the embedding table is under a third of the weights, and a multilingual vocabulary is never under a third.

Worth knowing about the speed

XNNPACK fp32 is slower than PyTorch eager here (40.9 ms against 32.9), and fp16 is slower again while halving the file — XNNPACK has no fp16 kernels for this graph and inserts casts instead. Core ML is the one that pays: 6.8 ms , five times eager, 100% delegated in a single subgraph. On iOS take the Core ML build; on Android the choice is fp32 for speed or fp16 for half the disk.

torch.export -> to_edge_transform_and_lower(partitioner) -> .pte (conversion scripts: executorch-models )