mlboydaisuke / multilingual-e5-large-ExecuTorch

multilingual-e5-large — ExecuTorch

Multilingual sentence embeddings: text in any of about a hundred languages in, one 1024-dimensional vector out, comparable across languages. For search and retrieval that never leaves the device.

• Source : intfloat/multilingual-e5-large — 560M parameters, 24 XLM-RoBERTa layers, 250k vocabulary, 1024-dimensional output
• License : MIT
• Input : input_ids and attention_mask , both [1, 256] int64
• Output : [1, 1024] , mean-pooled over the mask and L2-normalised inside the graph

Two parts of the recipe, and only one of them is in the graph

The pooling is in. sentence-transformers keeps it per model, and the six embedding models on this shelf do not agree:

Read off modules.json and 1_Pooling/config.json rather than assumed from the family name — paraphrase-multilingual is the same architecture family with the same pooling, and it does not normalise.

The prefix is not. E5 is trained with "query: " in front of a search query and "passage: " in front of a document, and it expects them at inference:

query:   query: how do I keep data on the phone?
passage: passage: On-device inference keeps the data on the phone.

That is text, so it happens before tokenisation and the .pte never sees it as anything but tokens. Leaving it out does not throw and does not look wrong — it returns a plausible vector that retrieves worse. The conversion repo's checker applies it, so the numbers below are for the recipe as the model intends it.

Verification

Mac arm64, median of 10, one 256-token sequence — a reference point for relative cost, not a device number. Eager fp32 on the same input is 101.1 ms. Cosine is measured against the model run in eager through its own pooling, over eight sentences.

And that the vectors are useful , which agreement alone cannot show. A paraphrase against an unrelated sentence, and then the same test across languages — "機械学習のモデルを端末の上で動かす" against "On-device inference keeps the data on the phone":

same language:   0.852 same meaning  vs  0.661 unrelated
across languages: 0.833 same meaning  vs  0.739 unrelated

The cross-lingual row is what this model is for, and it is the row the English-only models on this shelf fail. Note the scale: E5 puts everything high, so 0.70 for an unrelated pair is normal and the gap is what carries the signal, not the absolute number.

python convert/check_embed.py multilingual_e5_large fp32     # or fp16, int8, coreml

Not shipped

int8 converts and holds — worst cosine 0.997435, and it still separates the pairs — but it comes out at 1330.7 MB against fp16's 1118.3 MB . Dynamic int8 quantises the linear weights and leaves the token embedding table alone, and with a 250k vocabulary at 1024 dimensions that table is 1024 MB of the 2235.7 MB model , 46% of it.

The base model is the same story with a different ratio, and the pair is worth reading together: its table is 69% of the file and int8 lands 54% above fp16; here the table is 46% and int8 lands 19% above. The depth doubled and the vocabulary did not. This shelf's rule of thumb — int8 beats fp16 only when the embedding table is under about a third of the weights — holds at both points, and large is closer to the crossing.

Worth knowing about the speed

XNNPACK fp32 is level with PyTorch eager here (106.5 ms against 101.1), and fp16 is twice as slow while halving the file — XNNPACK has no fp16 kernels for this graph and inserts casts instead. Core ML is the one that pays: 27.2 ms , roughly four times eager, 100% delegated in a single subgraph. On iOS take the Core ML build; on Android the choice is fp32 for speed or fp16 for half the disk.

torch.export -> to_edge_transform_and_lower(partitioner) -> .pte (conversion scripts: executorch-models )