mlboydaisuke / paraphrase-multilingual-MiniLM-L12-v2-ExecuTorch

paraphrase-multilingual-MiniLM-L12-v2 — ExecuTorch

Sentence embeddings: text in, one vector out. For search, clustering and retrieval that never leaves the device.

• Source : sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 — 118M parameters, 12 BERT layers, 384-dimensional output
• License : Apache-2.0
• Input : input_ids and attention_mask , both [1, 256] int64
• Output : [1, 384]

The pooling is in the graph, on purpose

sentence-transformers keeps the recipe per model in 1_Pooling/config.json and modules.json , and the three small models on this shelf do not agree:

This one uses mean over the attention mask, and no normalisation. Getting it wrong does not throw: mean-pooling BGE, or normalising the multilingual one, gives vectors that look fine and rank wrong. So it is baked in rather than left to the caller.

Verification (Mac arm64, 2026-08-23)

Eager fp32 on the same input is 16.8 ms. Cosine is against the model run in eager through its own documented pooling, over eight sentences including one in Japanese — not against random token ids, which would tell you nothing.

And that the vectors are useful , which agreement alone cannot show:

0.511 a paraphrase   vs -0.135 an unrelated sentence

Across languages — "機械学習のモデルを端末の上で動かす" against "On-device inference keeps the data on the phone", with an unrelated sentence about the weather — this model scores 0.265 against -0.046 — it does .

Not shipped

int8 converts and holds — worst cosine 0.999610 against eager over the same eight sentences, cross-lingual pair included — but it comes out at 406.7 MB against fp16's 235.3 MB , so nothing would pick it. This model shows the reason most starkly: dynamic int8 quantizes the linear weights and leaves the token embedding table alone, and with a 250,037-token vocabulary that table is 384.1 MB of a 470.2 MB model — 82% of the weights. Quantizing every linear touches the other 18%. fp16 halves all of it.

An earlier version of this card said int8 does not export , with an IndexError: tensors used as indices must be long, int, byte or bool tensors . That was true of the build at the time and the cause was misread. The error is not in the embedding lookup: XNNPACKQuantizer's transform_for_annotation lifts every scalar argument of add.Tensor / mul.Tensor into a buffer and writes it as torch.tensor(float(arg)) regardless of the node's dtype, so the arange(n) + 0 that builds a HuggingFace attention mask comes back float32 and can no longer index. It is one line in ExecuTorch ( backends/xnnpack/quantizer/xnnpack_quantizer_utils.py , still present on main), and it affects every BERT-family model, not these three.

Worth knowing about the speed

XNNPACK fp32 is slower than PyTorch eager here (28.6 ms against 16.8), and fp16 is slower again while halving the file. Core ML is the one that pays: 3.9 ms, roughly 4.3x eager, at half the size. If this is going on an Apple device, take the Core ML build.

Conversion

python convert/export_embed.py paraphrase_multilingual
python convert/check_embed.py paraphrase_multilingual

Sequence length is fixed at 256; the attention mask makes padding harmless for mean pooling, and chunking anything longer is the caller's job.

(conversion scripts: executorch-models )