mlboydaisuke / all-MiniLM-L12-v2-ExecuTorch

all-MiniLM-L12-v2 — ExecuTorch

Sentence embeddings: text in, one vector out. For search, clustering and retrieval that never leaves the device. The deeper sibling of all-MiniLM-L6-v2 — twelve layers instead of six, the same 384-dimensional output and the same recipe.

• Source : sentence-transformers/all-MiniLM-L12-v2 — 33.4M parameters, 12 BERT layers, 384-dimensional output
• License : Apache-2.0
• Input : input_ids and attention_mask , both [1, 256] int64
• Output : [1, 384] , mean-pooled over the mask and L2-normalised inside the graph

The pooling is in the graph, on purpose

sentence-transformers keeps the recipe per model in 1_Pooling/config.json and modules.json , and the six embedding models on this shelf do not agree:

This one was read off the repo rather than assumed from the family name: modules.json lists Transformer, Pooling, Normalize and the pooling config sets pooling_mode_mean_tokens . Getting it wrong does not throw — mean-pooling BGE, or normalising the multilingual one, gives vectors that look fine and rank wrong — so it is baked in rather than left to the caller.

Verification

Mac arm64, median of 10, one 256-token sequence — a reference point for relative cost, not a device number. Eager fp32 on the same input is 17.4 ms. Cosine is measured against the model run in eager through its own documented pooling, over eight sentences including one in Japanese — not against random token ids, which would tell you nothing.

And that the vectors are useful , which agreement alone cannot show. A paraphrase against an unrelated sentence:

0.606 same meaning   vs  -0.110 unrelated

Across languages — "機械学習のモデルを端末の上で動かす" against "On-device inference keeps the data on the phone" — it scores -0.089 , below its score for an unrelated English sentence. That is the right answer for an English-only model, and the reason paraphrase-multilingual is on the shelf next to it.

python convert/check_embed.py all_minilm_l12 fp32     # or fp16, int8, coreml

Not shipped

int8 converts and holds — worst cosine 0.998275 against eager over the same eight sentences — but it comes out at 69.5 MB against fp16's 66.7 MB , so nothing would pick it. The arithmetic is the same as on L6: dynamic int8 quantises the linear weights and leaves the token embedding table alone, and that table is 46.9 MB of the 133.0 MB model. fp16 halves the table too. On a model whose weights are substantially a vocabulary, fp16 is the smaller build — the depth doubled from L6 and the table did not, which is why the gap here (69.5 against 66.7) is narrower than L6's (58.6 against 45.3).

Worth knowing about the speed

XNNPACK fp32 is slower than PyTorch eager here (29.2 ms against 17.4), and fp16 is slower again while halving the file — XNNPACK has no fp16 kernels for this graph and inserts casts instead. Core ML is the one that pays: 3.6 ms , roughly five times eager, 100% delegated in a single subgraph. On iOS take the Core ML build; on Android the portable file is fp32 unless the size matters more than the latency.

torch.export -> to_edge_transform_and_lower(partitioner) -> .pte (conversion scripts: executorch-models )