litert-community / OLMo-2-1B-Instruct

OLMo-2-1B-Instruct — LiteRT-LM (blockwise int4)

allenai/OLMo-2-0425-1B-Instruct converted to the LiteRT-LM ( .litertlm ) format for on-device inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models).

OLMo-2 is AllenAI's fully-open model family (Apache-2.0; open weights, data, and training code). This 1B variant is small enough to run on a phone — verified on iPhone 17 Pro. Converted with the official upstream litert-torch — no fork.

Usage

Run with the LiteRT-LM runtime:

litert_lm_main \
  --model_path OLMo-2-1B-Instruct_q4_block32_ekv4096.litertlm \
  --backend gpu \
  --input_prompt "Explain on-device AI in one sentence."

The .litertlm bundle carries the tokenizer and the prompt template (OLMo-2's native Tülu format — <|user|> / <|assistant|> , stop token <|endoftext|> ), so no separate tokenizer files are needed.

Run on Android

Update (July 2026): Google AI Edge Gallery v1.0.16+ can import litert-lm models directly from Hugging Face inside the app (tap + ) — no computer or adb needed. The manual steps below are only required on older builds or for sideloading a local file.

The easiest way to try this model on a phone is the official Google AI Edge Gallery app:

1. Install a recent Gallery (package com.google.ai.edge.gallery , APK from the repo's releases — 1.0.15+ supports .litertlm ).
2. Download OLMo-2-1B-Instruct_q4_block32_ekv4096.litertlm and push it to the device:

   adb push OLMo-2-1B-Instruct_q4_block32_ekv4096.litertlm /sdcard/Download/
   

3. In the app, tap + (bottom-right), pick the file, and choose CPU or GPU. At ~0.93 GB this 1B fits comfortably on an 8 GB phone.
4. Chat — the bundle already carries the tokenizer and OLMo-2 prompt template.

See the Gallery Importing Local Models guide for details. To embed it in your own Android app, use the LiteRT-LM Kotlin API ( com.google.ai.edge.litertlm:litertlm-android ).

Run on desktop (LiteRT-LM CLI)

The same .litertlm bundle runs on macOS / Linux / Windows with the official LiteRT-LM CLI — including as a local OpenAI-compatible API server :

pip install litert-lm
litert-lm import --from-huggingface-repo litert-community/OLMo-2-1B-Instruct OLMo-2-1B-Instruct_q4_block32_ekv4096.litertlm olmo-2-1b-instruct
litert-lm run olmo-2-1b-instruct     # interactive chat in the terminal
litert-lm serve           # local OpenAI-compatible API server

Quality — GSM8K

Measured on GSM8K (n=100, greedy, 0-shot chain-of-thought, identical prompt and answer-extraction for every row).

63 % is a strong, coherent, non-degenerate score for a 1B (the \boxed{} -style answers terminate cleanly at <|endoftext|> ). At 1B, 4-bit quantization costs ~9 pt vs bf16 — a small model has less redundancy to absorb int4 rounding than a 3B+ (where the same recipe is at parity). An int8 build recovers only ~2 pt (65 %) for +60 % size, so int4 is shipped as the best size/quality trade-off for on-device.

Conversion

Converted with the official upstream litert-torch export_hf (clean git worktree at upstream/main , dev-fork patches excluded). Olmo2ForCausalLM rides the stock converter with no custom code: QK-norm and OLMo-2's reordered post-norm lower to generic ops. The int4 recipe is blockwise (block 32) + OCTAV with the embedding at INT8.

Training data & PII

This is a weights-exact format conversion of allenai/OLMo-2-0425-1B-Instruct ; no new training was performed. OLMo-2 is fully open : pretrained on AllenAI's public Dolma -family corpus (web text, code, academic papers, encyclopedic and Q&A data) and post-trained with the public Tülu 3 SFT + preference mix. Because the pretraining corpus is web-derived it may incidentally contain names and other PII; none was deliberately collected and this format conversion adds none. Apply your own content/PII filtering before deployment. See the base model card for full dataset and safety details.

License

Apache-2.0, inherited from the base model allenai/OLMo-2-0425-1B-Instruct .