litert-community / LFM2.5-Embedding-350M

LFM2.5-Embedding-350M — LiteRT

LiquidAI/LFM2.5-Embedding-350M converted to LiteRT ( .tflite ) for on-device inference. A multilingual (11 languages) dense bi-encoder on the LFM2 hybrid backbone (gated short-convolutions + grouped-query attention) — one 1024-d vector per text, for retrieval, semantic search and RAG, fully offline on CPU.

Signatures

All signatures take batch-1, right-padded static shapes: input_ids int32 [1, S] , attention_mask int32 [1, S] (1 = real token, 0 = pad).

Pick the smallest signature that fits your token count. The vector does not depend on which one you pick : padding is zeroed inside the graph before the short-convolution and masked in attention, so the same text through embed_64 and through embed_512 returns a bitwise identical vector — verified on Mac and again on-device. Model max length is 512 tokens.

Contract

Two things must match the upstream model or results silently degrade:

• Asymmetric prompts, with the trailing space. Prefix queries with query: and passages with document: , exactly as stored in the base repo's config_sentence_transformers.json .
• CLS pooling and normalization are already in the graph. The output is the position-0 token vector, L2-normalized, so a plain dot product is cosine similarity. Do not pool or normalize again.

Quality

Measured against the PyTorch fp32 reference ( Lfm2BidirectionalModel + CLS + normalize) on the same inputs. fp16 is bitwise identical to the reference on every task metric below.

int8 costs 0.005 nDCG@10 and is a rounding difference on STS17. Per-vector agreement with the reference is cosine 0.9945–0.9955 for int8 and ≥0.9999999 for fp16.

Graph mechanics on the shipped artifacts, all three variants: cross-signature max|diff| 0.0 at 64/128/256/512, pad-content invariance 0.0 (garbage in the padded region cannot move the vector), and a live attention_mask (shortening it moves the output by 4.8e-02).

Speed

Apple-silicon Mac, XNNPACK, 16 threads, 20 iterations after warmup:

fp16 is slower than fp32 on CPU: XNNPACK unpacks fp16 weights to fp32 at run time, so the smaller file buys disk, not latency.

iPhone 17 Pro, XNNPACK, 6 threads (warm, after the first call on each signature):

The int8 model reproduces the Mac outputs bit-exactly on device across all tested cases (en/ja/de/ar/hi, short and long, both prompt forms). The fp16 model does run on an iPhone 17 Pro, but its 5.8 GiB peak is a function of that device's memory — treat fp16 as the desktop artifact. Its first invocation on each signature costs 355–538 ms of one-time XNNPACK fp16→fp32 unpacking, excluded from the warm figures above.

Usage (Python)

import numpy as np
from ai_edge_litert.interpreter import Interpreter
from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
it = Interpreter(model_path="LFM2.5-Embedding-350M_wi8fc.tflite")
embed = it.get_signature_runner("embed_128")

def encode(text, prompt):           # prompt = "query: " or "document: "
    ids = tok.encode(prompt + text).ids[:128]
    x = np.zeros((1, 128), np.int32)
    m = np.zeros((1, 128), np.int32)
    x[0, :len(ids)] = ids
    m[0, :len(ids)] = 1
    return list(embed(input_ids=x, attention_mask=m).values())[0][0]  # [1024]

q = encode("What is the capital of France?", "query: ")
d = encode("Paris is the capital and largest city of France.", "document: ")
print(float(q @ d))                 # already normalized -> dot product is cosine

On Android/iOS use the LiteRT runtime's SignatureRunner APIs with the same signature names; the tokenizer is the standard Hugging Face tokenizer.json (works with the tokenizers libraries for Rust/Swift/Kotlin).

License

LFM Open License v1.0 (see LICENSE , unchanged from the base model). Note the license's commercial-use threshold (Section 5). This repository redistributes converted Derivative Works of LiquidAI/LFM2.5-Embedding-350M with modification notices per Section 4; all credit for the model to Liquid AI .