litert-community / LFM2.5-2.6B

LFM2.5-2.6B — LiteRT-LM

LiquidAI/LFM2.5-2.6B converted to the LiteRT-LM ( .litertlm ) format for on-device inference with Google's LiteRT-LM runtime (requires litert-lm ≥ 0.14 ; the bundled executor-metadata section makes the same file run on 0.15's new state binding as well).

LFM2.5-2.6B is the flagship of Liquid AI's LFM2.5 on-device family: a hybrid architecture (22 gated short-convolution blocks + 8 grouped-query attention layers) and a thinking model — it reasons inside a <think>…</think> block before answering. The bundle declares the thought channel, so LiteRT-LM apps receive the reasoning and the answer as separate streams.

Quality — GSM8K

GSM8K, greedy, 0-shot chain-of-thought, max-tokens 2048 (thinking model), n=100, same harness for all rows:

Both files pass an 8-question sanity gate 8/8 with zero degenerate outputs (CPU), a 42-length prefill sweep with zero corrupt first tokens (fresh engine per length), and a 3-turn conversation gate (fact recall + arithmetic).

Usage

pip install litert-lm
litert-lm run ./LFM2.5-2.6B_int4.litertlm --prompt "What is 17 + 25?"

The model thinks before answering — through the Python/C++ API the reasoning arrives on the thought channel and the reply stays clean; the CLI prints both.

--max-num-tokens tip: reasoning turns are long — give the model at least 2048 tokens of budget for math/complex questions, or the think block may not close. Decode speed drops as the token budget grows, so don't set it higher than you need.

Multi-turn note: on the current runtime the conversation context retains previous turns' reasoning, so long chats fill the context faster than the visible text suggests. For unrelated questions, prefer fresh conversations.

Speed

litert-lm benchmark , CPU backend, Mac M4 Max, max-num-tokens 1024 (prefill-1024 rows measured at max-num-tokens 2048):

GPU backend ( litert-lm benchmark … --backend gpu --cache no , litert-lm 0.16.0, same machine; both variants verified to generate correct text on GPU before quoting):

On Android the same bundle runs GPU-accelerated with litert-lm ≥ 0.16.0. Pixel 8a (Tensor G3), litert_lm_main v0.16.0 built from the release tag, 261-token prompt with generation capped at 256, 3 runs per backend (int4 file):

At this size the GPU wins across the board on-device — ~3–5× prefill, ~3–5× faster to the first token, and a steadier decode (the CPU rows spread because they throttle; the GPU rows repeat to within 1%) — so prefer the GPU backend where the runtime offers it. A short prompt shifts the ratio further toward the GPU: with a 19-token prompt the same file prefills at 37.7–38.8 tok/s on GPU against 4.1 tok/s on CPU.

iPhone 17 Pro (CPU backend, cold first runs): the int4 file answers the 8-question quality gate 8/8 and decodes at ~20 tok/s warm (the first, cold run is slower while caches build). The int8 file also loads and runs on iPhone 17 Pro when the host app carries the extended-virtual-addressing / increased-memory entitlements (its 2.87 GB single weight section exceeds what a default-entitlement app will memory-map) — int4 is the recommended phone variant (smaller, faster on phone-class memory bandwidth).

Run on Android

Install a recent Google AI Edge Gallery (1.0.16+ imports .litertlm directly from Hugging Face), import this repo (or adb push a file and use local import), select the CPU backend, and chat. GPU works from litert-lm ≥ 0.16.0 ( litert_lm_main --backend=gpu , or the SDKs); the Gallery app bundles its own runtime which may lag behind that version — if its GPU toggle fails, use CPU there.

Conversion notes

Converted with released litert-torch 0.9.2 (upstream lfm2 hybrid support incl. the ShortConv prefill-pad fix) and packaged for litert-lm ≥ 0.14 (the executor-metadata section binds the 22 conv states + 16 KV caches; files exported without it do not run on 0.15).

• The generation prompt pre-fills <think> exactly as the vendor chat template does. This matters: with a bare assistant prompt, think-block emission becomes the model's choice, and the int4 quantization degrades that discipline first (unscaffolded rambling in place of answers on multi-turn). With the pre-fill, all variants answer tersely and the thought channel routes cleanly.
• The checkpoint is notably sparse: OCTAV int4-b32 produced 746k all-zero weight blocks whose zero scales the XNNPACK delegate rejects at load. The zero scales are patched to the tensor's smallest nonzero scale (dequantization unchanged — the blocks are all zeros).
• int8 quantizes convs at export time (safe); post-hoc conv int8 breaks generation. int4 keeps convs float.
• Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks; a 42-length first-token sweep verifies no prefill-padding state corruption.