litert-community / SmolLM3-3B

SmolLM3-3B — LiteRT-LM (blockwise int4)

HuggingFaceTB/SmolLM3-3B converted to the LiteRT-LM ( .litertlm ) format for on-device inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models).

SmolLM3 is a fully-open 3B decoder (Apache-2.0) with GQA, a NoPE attention schedule, multilingual support, and long-context training — a strong small reasoner.

Usage

Run with the LiteRT-LM runtime:

# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
litert_lm_main \
  --model_path SmolLM3-3B_q4_block32_ekv4096.litertlm \
  --backend gpu \
  --input_prompt "Explain on-device AI in one sentence."

The .litertlm bundle carries the tokenizer and the prompt template (ChatML — <|im_start|>role / <|im_end|> , stop token <|im_end|> ), so no separate tokenizer files are needed.

Run on Android

Update (July 2026): Google AI Edge Gallery v1.0.16+ can import litert-lm models directly from Hugging Face inside the app (tap + ) — no computer or adb needed. The manual steps below are only required on older builds or for sideloading a local file.

The easiest way to try this model on a phone is the official Google AI Edge Gallery app — it runs .litertlm models fully on-device and can import your own:

1. Install a recent Gallery (package com.google.ai.edge.gallery , APK from the repo's releases — 1.0.15+ supports .litertlm ). Older 1.0.x builds (package com.google.aiedge.gallery ) only accept the legacy MediaPipe .task format and reject .litertlm .
2. Download SmolLM3-3B_q4_block32_ekv4096.litertlm from this repo and push it to the device:

   adb push SmolLM3-3B_q4_block32_ekv4096.litertlm /sdcard/Download/
   

3. In the app, tap the + button (bottom-right), pick the file, and choose the GPU backend (CPU also works).
4. Chat. Nothing else to configure — the .litertlm bundle already carries the tokenizer and ChatML prompt template.

See the Gallery Importing Local Models guide for details. To embed the model in your own Android app instead, use the LiteRT-LM Kotlin API (Gradle artifact com.google.ai.edge.litertlm:litertlm-android , getting started ).

Run on desktop (LiteRT-LM CLI)

The same .litertlm bundle runs on macOS / Linux / Windows with the official LiteRT-LM CLI — including as a local OpenAI-compatible API server :

pip install litert-lm
litert-lm import --from-huggingface-repo litert-community/SmolLM3-3B SmolLM3-3B_q4_block32_ekv4096.litertlm smollm3-3b
litert-lm run smollm3-3b     # interactive chat in the terminal
litert-lm serve           # local OpenAI-compatible API server

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine.

Reproducibility: the GPU rows repeat to within about 1% across invocations; the CPU rows are noisier — re-running the 1B control six times spread its CPU decode over 29.0–33.3 tok/s, so treat the CPU column as accurate to roughly ±7%.

The iPhone row is one cold run through the LiteRTDemo harness on iOS 27.0 (prompt "Explain on-device AI in one short sentence.", 512-token budget, no warm-up turn), read back from its run log. Its prefill figure is measured on that short prompt, so it reflects fixed per-turn overhead rather than prefill throughput and is not comparable to the 256-token desktop column.

Accuracy note

Measured on GSM8K (n=100, greedy, 0-shot chain-of-thought asking for #### <n> , identical prompt and answer-extraction for both rows — only the quantization differs).

LiteRT int4 is fully at parity — 0.0 pt vs the bf16 reference. The blockwise-32 + OCTAV recipe with a 4096 KV cache preserves reasoning accuracy exactly at n=100. The model produces visible step-by-step chain-of-thought in the answer body and terminates cleanly at <|im_end|> (no rambling).

Conversion

Converted with litert-torch via its generic export_hf path. SmolLM3ForCausalLM rides the existing converter with no custom code: the NoPE attention schedule (rotary disabled on every 4th layer, no_rope_layer_interval=4 ) lowers to generic ops with no custom kernel. The int4 recipe is blockwise (block 32) + OCTAV optimal-clipping with the embedding kept at INT8; the embedding is externalized into its own bundle section so the main weights section stays under the iOS ~2 GiB single-mmap limit. Blockwise (not channelwise) int4 plus OCTAV is what holds reasoning accuracy at parity.

Training data & PII

This is a weights-exact format conversion of HuggingFaceTB/SmolLM3-3B ; no new training was performed. SmolLM3 was trained by Hugging Face on ~11T tokens of publicly documented data — web (FineWeb-Edu, DCLM), code (StarCoder-family), math, and multilingual sources — plus public SFT/preference sets. Being web-derived it may incidentally contain PII; none was deliberately collected and this format conversion adds none. Apply your own content/PII filtering before deployment. See the base model card for the full data mixture.

License

Apache-2.0, inherited from the base model HuggingFaceTB/SmolLM3-3B .