mlboydaisuke / MiniCPM5-1B-CoreAI

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export ) into .aimodel bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol ( apple-silicon-llm-bench , macOS 27 beta 26A5353q, 2026-06-11).

This model has no row on DeviceMark , the on-device LLM leaderboard.

MiniCPM5-1B — Core AI (int8 block-32, runs on iPhone)

Apple Core AI ( .aimodel ) conversion of openbmb/MiniCPM5-1B — OpenBMB's 1.08B on-device LLM with hybrid Think / No-Think reasoning and 128K context, reaching 1B-class open-source SOTA. Runs fully on-device on iPhone and Apple Silicon Macs (GPU, pipelined engine).

Revision note (2026-09-09). This revision replaces the per-channel int8 bundle published as 5ad650f . That bundle's LM head had dead rows from vocab id ~65024 up — every token there scored ~0 in the engine, <|im_end|> (130073) included — so a chat turn never halted (it ran to the token cap) and any answer needing a high-id token lost it. The card's "halts cleanly" line was wrong. Details and the measurements are under Why per-block-32 below; pin this revision or newer.

Part of the community Core AI model zoo: https://github.com/john-rocky/coreai-model-zoo

Use it

⚡ One line — run the kit's task op on this model ( import CoreAIOps ; no session, no model plumbing, downloads on first use):

let tldr = try await CoreAI.summarize(text, options: .model("minicpm5-1b"))

Every op, one shape — Cookbook .

▶️ Run it (source) — the ChatDemo runner (GUI + CLI, one app for every chat model in the catalog):

git clone https://github.com/john-rocky/coreai-kit
open coreai-kit/Examples/ChatDemo/ChatDemo.xcodeproj
# → Run, then pick "MiniCPM5 1B" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/ChatDemo
swift run chat-cli --model minicpm5-1b --prompt "What can you do, offline?"

💻 Build with it — complete; the glue is kit API, copy-paste runs:

import CoreAIKit

let chat = try await ChatSession(catalog: "minicpm5-1b")
let reply = try await chat.respond(to: prompt)
// reply: the answer, generated fully on-device

The take-home is Examples/ChatDemo/Sources/QuickStart.swift — this exact code as one typed function, no UI; the CLI is an argument shell over it, and the GUI drives the same ChatSession across turns for its transcript. Multi-turn? Hold the ChatSession and call respond(to:) per turn — it keeps the conversation history; streamResponse(to:) yields tokens as they decode.

Integration checklist

• SPM: https://github.com/john-rocky/coreai-kit → product CoreAIKit
• Info.plist: none needed
• Entitlements: none needed
• First run downloads the model — 1.1 GB (Mac) / 1.1 GB (iPhone) — then it loads from the local cache (Application Support; progress via the downloadProgress callback)
• Measure in Release — Debug is ~3× slower on per-token host work

Neural Engine bundle (iOS static export, AOT h18p) — 2026-09-15

ios-ane-h18p/ is Apple's stock coreai.llm.export --platform iOS static export of this checkpoint (k-means 8-bit palettized, group 32 ( conversion/minicpm5_pal8_g32.yaml ); embeddings int8; static graphs prompt_opt / extend × contexts {256, 512, 1024, 2048, 4096} × query {8, 16, 64}), AOT-compiled with xcrun coreai-build compile --platform iOS --preferred-compute neural-engine --architecture h18p (31/31 ANE regions, 1.3 GB). It loads through Apple's EngineFactory → StaticShapeEngine unchanged (iPhone 17-class devices). ios-static/ is the same export before AOT — the portable IR; compile it for another chip yourself. Apple's default 4-bit preset fails the same gate on this checkpoint (two margin-clear flips on a chat turn, unchanged with fp16 embeddings; gate-minicpm5-1b-ane-device-4bit-FAIL.json ), so the 1B ships at 8 bits.

Gated on an iPhone 17 Pro (iOS 27.0) against the fp32 HF oracle: teacher-forced single-step sweep 24/24 + 10/10 + 16/16 and free-run greedy token-exact including the stop — transcript in the zoo, models/minicpm5-1b/gate-minicpm5-1b-ane-device.json . Speed on the same phone (Apple llm-benchmark method, 512-token prompt, 1024 generated, 5 trials, two back-to-back runs): decode 69.6 / 58.6 tok/s , prefill 2710 / 2364 tok/s; footprint 1.3 GB (p128 / g256: decode 62.3). The Neural Engine is the power-efficient lane; these are not a GPU comparison (different protocol from the int8/ row above).

Same-day interleaved A/B on the same phone (one app embedding both bundles, 128-token prompt, 256 generated, 5 trials, ANE-GPU-ANE-GPU): Neural Engine 8-bit 76.8 / 76.8 tok/s decode (3899 / 3865 prefill) vs this repo's int8/ GPU bundle 67.1 / 64.4 (2359 / 2270) — equal weight bytes per token, so +17 % decode and +70 % prefill on the ANE; GPU cold specialization 3.6 s vs ANE AOT load 0.2 s.

Measured

Halt check through the engine on the same phone-shaped bundle: the Think-mode turn 1+1=? stops on its own after 171 tokens (cap 400); the no-think turn after 6.

⚠️ iPhone context cap: prompt + generated tokens must stay under 1024. The bundle declares a 131072 dynamic KV, and the shipped CoreAIPipelinedEngine caps iOS growing-KV capacity at 1024 (its guard against the iOS compiler miscompiling growing-KV specializations at seq ≥ 2048). Chunk or trim the history on iOS; macOS has no cap.

Why per-block-32 (what was wrong with the previous revision)

The previous bundle ( 5ad650f , int8 per-channel absmax) passed a 24-token greedy parity check and still never ended a chat turn. Teacher-forcing it through the engine against the fp32 reference located the defect in the LM head, by vocab id:

Every probed row at id ≥ 65039 is dead in the per-channel bundle and every probed row ≤ 65023 is healthy (16 probes; the boundary lies in that 16-id window, which contains 65024 = 127 × 512). Rows below it match fp32 to ~0.01 in probability, which is why free-running English prose looked fine. A fresh per-channel export on the same toolchain (coreai-torch 0.4.1 / coreai-opt 0.2.1 / coreai-core 1.0.0b2) reproduces the shipped bundle's logits to four decimals, so this is a property of the per-channel int8 path for this 130560-row head, not a one-off; per-block-32 (this bundle), the CLI's default int4 preset and fp16 ( --compression none ) are all clean on the same probes. Which component owns the row cut-off (quantizer, converter or the runtime's per-channel int8 matmul) is not established here.

The gate that catches it is now part of the zoo's cli/coreai_verify.py : --chat no-think --prompt "1+1=?" makes the fp32 oracle's <|im_end|> a gated step — a bundle that runs past a stop the oracle takes at margin 0.80 fails — and --must-stop-within N is the plain halt check. The old bundle is red on both, this one green.

Quantization

Weight-only symmetric int8, per-block-32 (a scale per 32-wide block along the input dim; no clipping), applied as a torch pre-export pass via coreai-opt ; SDPA / RoPE / RMSNorm stay full precision — the same YAML that ships the 2B .

uv run coreai.llm.export openbmb/MiniCPM5-1B --experimental --compute-precision float16 \
  --compression-config minicpm5_int8sym_b32.yaml
# minicpm5_int8sym_b32.yaml: quantization_config → op_state_spec.weight = {dtype: int8,
#   qscheme: symmetric, granularity: {type: per_block, block_size: 32}}

Conversion notes

• llama → mistral remap. MiniCPM5-1B's model_type is llama ; the stock exporter has no llama graph family, but Mistral's builder is architecturally identical for this config (GQA, no qkv bias, no qk-norm, explicit head_dim honored). One-line remap in the model registry.
• Chat EOS. Base eos_token is </s> , but the chat template ends turns with <|im_end|> (id 130073). The bundle's tokenizer eos_token is set to <|im_end|> (as Qwen ships). Checked through the engine with the chat template applied: the no-think turn 1+1=? answers 1+1=2 and stops at step 5, where the fp32 reference stops (margin 0.80).
• Dynamic-shape bundle → the Core AI pipelined engine (the iPhone path); a static iOS export routes to the static-shape engine instead, which this FM-format bundle doesn't target.
• Thinking. The model thinks by default ( <think>…</think> before the answer); pass enable_thinking=False through the chat template for a direct answer. Give generation a generous budget (the kit caps at 4096) — the think trace alone can run a few hundred tokens.

Run

// iOS / macOS, via Foundation Models
import FoundationModels
import CoreAILanguageModels
let model = try await CoreAILanguageModel(resourcesAt: modelURL)   // int8/ bundle
let session = LanguageModelSession(model: model)
print(try await session.respond(to: "Explain on-device AI in one sentence."))

Reproduce

Exporter, gate, card and port notes live in the Core AI model zoo : models/minicpm5-1b/ , conversion/export_minicpm5.py , knowledge/minicpm5-1b.md .

python3 conversion/zoo_convert.py show minicpm5-1b
python3 conversion/zoo_convert.py run  minicpm5-1b
python3 cli/coreai_verify.py <bundle> --chat no-think --prompt "1+1=?" -n 16 --must-stop-within 16

License

Apache-2.0 (upstream MiniCPM5 license). Model © OpenBMB — see https://huggingface.co/openbmb/MiniCPM5-1B . Conversion: community.

More models in this format: Core AI Model Zoo — 75 models, each with the recipe that produced it.

Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.