AXERA-TECH / MiniCPM-V-4.6

MiniCPM-V-4.6 on AXERA NPU

Ready-to-run AX650 deployment package for openbmb/MiniCPM-V-4.6 .

• Runtime: axllm serve and axllm run
• Target: AX650 / AX650N aarch64
• Text backend: Qwen3.5 dense LLM, bf16 , non-GPTQ weights
• Vision backend: fixed-shape 448x448 MiniCPM-V-4.6 vision encoder
• Context layout: prefill_len=128 , kv_cache_len=2047 , prefill_max_token_num=1280
• Validated capabilities in this release:
  • text-only chat
  • long prompt multi-chunk prefill
  • single-image understanding
  • video understanding
• Included board binary:
  • bin/axllm for AX650/AX650N aarch64

This package has been validated for text chat, single-image understanding, and video understanding through the OpenAI-compatible axllm serve API.

Supported Platform

• AX650 / AX650N

Performance and Footprint

Measurements below are from the preserved AX650 runtime log in Inference Log , plus a repeated text-only TTFT recheck on 2026-06-05 . TTFT means time to first token. The text-only smoke prompt is intentionally kept within one 128 -token prefill group. Its TTFT is the 5-run average for the prompt 1+1等于几？只输出数字。 ; the response is one token, so decode throughput is not reported for that row.

Flash Footprint

Startup Runtime Footprint

Measured on AX650 on 2026-06-05 by reading /proc/ax_proc/mem_cmm_info before startup and after the /health endpoint became ready. The measurement kept the existing GPTQ service on port 8000 running and records only the incremental CMM used by this extra service.

The preserved standalone inference log below records remain_cmm(6933 MB) for an earlier single-service startup; use the incremental CMM row above for the board-side footprint measured during this README refresh.

The runtime log also records the video sampling decision:

MiniCPM-V-4.6 video frames selected: 19/59 (configured_cap=0, tail_tokens=1278, max_tail=1280, precompute_len=0)

Repository Layout

.
├── README.md
├── bin/
│   ├── axllm
│   └── axllm.version.json
├── assets/
│   └── smoke_image.png
├── minicpm_v46_tokenizer/
├── python/
│   ├── infer_axmodel.py
│   ├── infer_torch.py
│   └── minicpm_v46_tokenizer/
├── minicpmv4_6_vision_448.axmodel
├── qwen3_5_text_p128_l0_together.axmodel
├── ...
├── qwen3_5_text_p128_l23_together.axmodel
├── qwen3_5_text_post.axmodel
├── model.embed_tokens.weight.bfloat16.bin
├── config.json
├── post_config.json
└── minicpm_v46_tokenizer.txt

Install axllm

Option 1: use the validated binary included in this repository:

chmod +x ./bin/axllm

Option 2: build or install the latest ax-llm yourself:

• Repo: https://github.com/AXERA-TECH/ax-llm
• The included binary metadata is recorded in bin/axllm.version.json .

Run on AX650

From the repository root on the board:

chmod +x ./bin/axllm
./bin/axllm serve . --port 18080

Expected model id:

AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047

Text Request

curl http://127.0.0.1:18080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "1+1等于几？只输出数字。"}
        ]
      }
    ],
    "max_tokens": 32
  }'

Image Request

python - <<'PY'
import base64
import json
from pathlib import Path
from urllib.request import Request, urlopen

img = Path("assets/smoke_image.png").read_bytes()
payload = {
    "model": "AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Please briefly describe this image."},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/png;base64," + base64.b64encode(img).decode()
                    },
                },
            ],
        }
    ],
    "max_tokens": 64,
}
req = Request(
    "http://127.0.0.1:18080/v1/chat/completions",
    data=json.dumps(payload).encode(),
    headers={"Content-Type": "application/json"},
)
with urlopen(req, timeout=60) as resp:
    print(resp.read().decode())
PY

Video Request

axllm serve supports a frames directory or a raw video file:

curl http://127.0.0.1:18080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "image_url", "image_url": {"url": "video:/path/to/frames_dir"}},
          {"type": "text", "text": "Describe this video."}
        ]
      }
    ],
    "max_tokens": 256
  }'

For a raw video file, use video:/path/to/video.mp4 or video:/path/to/video.mp4:2 to request a sampling FPS of 2 .

Interactive CLI

./bin/axllm run .

After each prompt:

• press Enter directly for text-only chat
• input an image path for single-image chat
• input video:<frames_dir> for video chat

Python Reference Scripts

Official PyTorch Reference

python/infer_torch.py is a developer-side reference script. It requires the full upstream HuggingFace model with weights and is intended for x86/GPU validation.

cd python
python infer_torch.py \
  --model-path /data/tmp/yongqiang/nfs/auto_model_deployment/Minicpm-V-4.6-hf-original/MiniCPM-V-4.6 \
  --prompt "1+1等于几？请直接回答。"

Python .axmodel Debug Runner

python/infer_axmodel.py is a text-only per-layer debug runner for the compiled LLM package:

cd python
python3 infer_axmodel.py \
  --hf-model ./minicpm_v46_tokenizer \
  --axmodel-dir .. \
  --mode generate \
  --prompt "1+1等于几？请直接回答。" \
  --prompt-mode prefill \
  --max-new-tokens 16 \
  --kv-cache-len 2047

Notes

1. .axmodel files are board-only. Do not run them on x86.
2. This package was compiled with --prefill_len 128 --kv_cache_len 2047 .
3. The AX650 build keeps FLOAT_MATMUL_USE_CONV_EU=1 enabled to reduce TTFT.
4. bin/axllm.version.json records the exact runtime binary commit and verification metadata.
5. The packaged vision encoder is fixed at 448x448 ; arbitrary image/video sizes are resized by the runtime preprocessor.
6. Video requests are isolated to the current user turn and drop KV cache after the response to avoid stale multimodal history reuse.

Inference Log

root@ax650 ~/yongqiang/auto_model_deployment/MiniCPM-V-4.6 # ./bin/axllm serve .
14:59:39.575 INF sidecar_string_list_value:251 | loaded layer_types[24] from ./minicpm_v46_tokenizer/config.json
14:59:39.577 INF Init:2239 | LLM init start
14:59:39.577 INF Init:2248 | mixed attention enabled: full_attention_interval=4 ref_full_layer_idx=3
14:59:39.577 INF Init:2270 | attention config: layers=24 sliding=0 full=6 linear=18 sliding_window=0 ref_full_layer_idx=3
tokenizer_type = 3
huggingface tokenizer mode = gpt2_byte_bpe
 44% | ##############                   |  12 /  27 [4.73s<10.63s, 2.54 count/s] init 10 axmodel ok,remain_cmm(7587 MB                                                                                                                       96% | ##############################   |  26 /  27 [6.15s<6.39s, 4.23 count/s] init post axmodel ok,remain_cmm(6933 MB)
14:59:45.729 INF Init:2420 | max_token_len : 2047
14:59:45.729 INF Init:2423 | kv_cache_size : 512, kv_cache_num: 2047
14:59:45.729 INF init_groups_from_model:1622 | prefill_token_num : 128
14:59:45.729 INF init_groups_from_model:1870 | decode grp: 0, gid: 0, max_token_len : 2047
14:59:45.729 INF init_groups_from_model:1874 | prefill grp: 0, gid: 1, history_cap: 0, total_cap: 128, symbolic_cap: 1
14:59:45.729 INF init_groups_from_model:1874 | prefill grp: 1, gid: 2, history_cap: 128, total_cap: 256, symbolic_cap: 128
14:59:45.729 INF init_groups_from_model:1874 | prefill grp: 2, gid: 3, history_cap: 256, total_cap: 384, symbolic_cap: 256
14:59:45.729 INF init_groups_from_model:1874 | prefill grp: 3, gid: 4, history_cap: 384, total_cap: 512, symbolic_cap: 384
14:59:45.729 INF init_groups_from_model:1874 | prefill grp: 4, gid: 5, history_cap: 512, total_cap: 640, symbolic_cap: 512
14:59:45.729 INF init_groups_from_model:1874 | prefill grp: 5, gid: 6, history_cap: 640, total_cap: 768, symbolic_cap: 640
14:59:45.729 INF init_groups_from_model:1874 | prefill grp: 6, gid: 7, history_cap: 768, total_cap: 896, symbolic_cap: 768
14:59:45.729 INF init_groups_from_model:1874 | prefill grp: 7, gid: 8, history_cap: 896, total_cap: 1024, symbolic_cap: 896
14:59:45.729 INF init_groups_from_model:1874 | prefill grp: 8, gid: 9, history_cap: 1024, total_cap: 1152, symbolic_cap: 1024
14:59:45.729 INF init_groups_from_model:1874 | prefill grp: 9, gid: 10, history_cap: 1152, total_cap: 1280, symbolic_cap: 1152
14:59:45.729 INF init_groups_from_model:1881 | prefill_max_token_num : 1280
14:59:45.729 INF init_layer_groups:1026 | layer 0 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.729 INF init_layer_groups:1026 | layer 1 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.729 INF init_layer_groups:1026 | layer 2 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.729 INF init_layer_groups:1026 | layer 4 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.729 INF init_layer_groups:1026 | layer 5 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.729 INF init_layer_groups:1026 | layer 6 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.729 INF init_layer_groups:1026 | layer 8 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.729 INF init_layer_groups:1026 | layer 9 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.729 INF init_layer_groups:1026 | layer 10 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.729 INF init_layer_groups:1026 | layer 12 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.729 INF init_layer_groups:1026 | layer 13 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.729 INF init_layer_groups:1026 | layer 14 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.730 INF init_layer_groups:1026 | layer 16 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.730 INF init_layer_groups:1026 | layer 17 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.730 INF init_layer_groups:1026 | layer 18 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.730 INF init_layer_groups:1026 | layer 20 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.730 INF init_layer_groups:1026 | layer 21 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.730 INF init_layer_groups:1026 | layer 22 prefill groups=2 ref=10, reuse gid 2 for later prefill chunks
14:59:45.730 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  27 /  27 [6.16s<6.16s, 4.39 count/s] embed_selector init ok
14:59:47.002 INF Init:1507 | MiniCPM-V-4.6 token ids: image_pad=248056 video_pad=248057
14:59:47.002 INF Init:1514 | VisionModule init ok: type=MiniCPMV46VL, tokens_per_block=64, embed_size=1024, out_dtype=fp32
14:59:47.002 WRN Init:1523 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)
14:59:47.004 INF load_config:444 | load config:
14:59:47.004 INF load_config:444 | {
14:59:47.004 INF load_config:444 |     "enable_repetition_penalty": false,
14:59:47.004 INF load_config:444 |     "enable_temperature": false,
14:59:47.004 INF load_config:444 |     "enable_top_k_sampling": false,
14:59:47.004 INF load_config:444 |     "enable_top_p_sampling": false,
14:59:47.004 INF load_config:444 |     "penalty_window": 20,
14:59:47.004 INF load_config:444 |     "repetition_penalty": 1.2,
14:59:47.004 INF load_config:444 |     "temperature": 0.9,
14:59:47.004 INF load_config:444 |     "top_k": 10,
14:59:47.004 INF load_config:444 |     "top_p": 0.8
14:59:47.004 INF load_config:444 | }
14:59:47.004 INF Init:2532 | LLM init ok
Starting server on port 8000 with model 'AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047'...
API URLs:
  GET  http://127.0.0.1:8000/health
  GET  http://127.0.0.1:8000/v1/models
  POST http://127.0.0.1:8000/v1/chat/completions
  GET  http://10.168.232.217:8000/health
  GET  http://10.168.232.217:8000/v1/models
  POST http://10.168.232.217:8000/v1/chat/completions
  GET  http://172.17.0.1:8000/health
  GET  http://172.17.0.1:8000/v1/models
  POST http://172.17.0.1:8000/v1/chat/completions
Aliases:
  GET  http://127.0.0.1:8000/models
  POST http://127.0.0.1:8000/chat/completions
  GET  http://10.168.232.217:8000/models
  POST http://10.168.232.217:8000/chat/completions
  GET  http://172.17.0.1:8000/models
  POST http://172.17.0.1:8000/chat/completions
OpenAI API Server starting on http://0.0.0.0:8000
Max concurrency: 1
Models: AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047
15:00:35.571 INF operator():2136 | OpenAI chat request: model=AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047 stream=1 max_tokens=2048 has_temperature=0 temperature=1.0000 has_top_p=0 top_p=1.0000 messages=1 stop=0
15:00:35.572 INF SetKVCache:3134 | decode_grpid:0 prefill_grpid:1 history_cap:0 total_cap:128 symbolic_cap:1 precompute_len:0 input_num_token:16 prefer_symbolic_group:0
15:00:35.572 INF SetKVCache:3156 | current prefill_max_token_num:1280
15:00:35.588 INF SetKVCache:3172 | first run
15:00:35.602 INF Run:3376 | input token num : 16, prefill_split_num : 1
15:00:35.602 INF Run:3387 | MiniCPM-V-4.6 short text prefill uses decode replay: input_tokens=16 prefill_tokens=128 precompute_len=0
15:00:35.602 INF Run:3471 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=16
15:00:36.091 INF Run:3886 | ttft: 488.83 ms
I'm a model from the MiniCPM series, developed by Modelbest and OpenBMB. For more details, you can visit https://github.com/OpenBMB/

15:00:37.997 NTC Run:4298 | hit eos,decode avg 17.84 token/s
15:00:37.997 INF GetKVCache:3087 | precompute_len:51, remaining:1229 (tracked)
15:01:11.728 INF operator():2136 | OpenAI chat request: model=AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047 stream=1 max_tokens=2048 has_temperature=0 temperature=1.0000 has_top_p=0 top_p=1.0000 messages=3 stop=0
15:01:11.941 WRN Run:4330 | video history is isolated to current user turn: old_history=3 new_history=1 old_media_inputs=1 new_media_inputs=1
15:01:12.096 INF extract_video_frames_ffmpeg:386 | Extracting raw video container to frames: /tmp/axllm_media/media_863804170273309_0.mp4 -> /tmp/axllm_video_frames/video_863804326631725_0 fps=1.9907760708716282
15:01:13.679 INF collect_video_frame_paths:466 | Video fps sampling: path=/tmp/axllm_media/media_863804170273309_0.mp4 fps=2 duration=30.139s target_frames=60 selected=59
15:01:13.687 INF Prepare:2594 | MiniCPM-V-4.6 video frames selected: 19/59 (configured_cap=0, tail_tokens=1278, max_tail=1280, precompute_len=0)
15:01:24.499 INF SetKVCache:3134 | decode_grpid:0 prefill_grpid:1 history_cap:0 total_cap:128 symbolic_cap:1 precompute_len:0 input_num_token:1278 prefer_symbolic_group:0
15:01:24.499 INF SetKVCache:3156 | current prefill_max_token_num:1280
15:01:24.516 INF SetKVCache:3172 | first run
15:01:24.531 INF Run:3376 | input token num : 1278, prefill_split_num : 10
15:01:24.531 INF Run:3471 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=128
15:01:24.689 INF Run:3471 | prefill chunk p=1 history_len=128 grpid=3 kv_cache_num=256 input_tokens=128
15:01:24.865 INF Run:3471 | prefill chunk p=2 history_len=256 grpid=4 kv_cache_num=384 input_tokens=128
15:01:25.041 INF Run:3471 | prefill chunk p=3 history_len=384 grpid=5 kv_cache_num=512 input_tokens=128
15:01:25.219 INF Run:3471 | prefill chunk p=4 history_len=512 grpid=6 kv_cache_num=640 input_tokens=128
15:01:25.398 INF Run:3471 | prefill chunk p=5 history_len=640 grpid=7 kv_cache_num=768 input_tokens=128
15:01:25.578 INF Run:3471 | prefill chunk p=6 history_len=768 grpid=8 kv_cache_num=896 input_tokens=128
15:01:25.759 INF Run:3471 | prefill chunk p=7 history_len=896 grpid=9 kv_cache_num=1024 input_tokens=128
15:01:25.943 INF Run:3471 | prefill chunk p=8 history_len=1024 grpid=10 kv_cache_num=1152 input_tokens=128
15:01:26.128 INF Run:3471 | prefill chunk p=9 history_len=1152 grpid=10 kv_cache_num=1152 input_tokens=126
15:01:26.327 INF Run:3886 | ttft: 1795.28 ms
好的，我很乐意为您描述这个视频的内容。

这张图片展示了一个活泼可爱的红熊猫，它正积极地参与着某种游戏或探索活动。从画面来看，这只红熊猫有着标志性的红棕色和黑色相间的毛色，面部和胸部有白色的标记，非常醒目。它正用前爪紧紧抓住一根由竹子或类似材料搭建的木制梯子或支架，身体微微前倾，似乎正在用力或试图攀爬。它的姿态充满了动感和专注，显示出它对这个环境非常感兴趣。

在画面的下方，还有一只较小的红熊猫，它正抬头向上看，似乎是在观察上方那只正在攀爬的同伴，或者是在等待自己的机会。这两只红熊猫被安置在一个绿色的草地上，背景中可以看到一些绿色的围栏或墙壁，暗示这可能是在一个动物园或野生动物保护区的围栏内。

整个场景充满了自然和活力，红熊猫们通过互动和探索，展现了它们的天性和对环境的适应。这个视频捕捉到了它们 playful 和 curious 的瞬间，让人感受到它们活泼可爱的性格。

15:01:37.610 NTC Run:4298 | hit eos,decode avg 18.79 token/s
15:01:37.610 WRN Run:4631 | drop KV cache after isolated video-history request
^@15:02:51.406 INF operator():2136 | OpenAI chat request: model=AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047 stream=1 max_tokens=2048 has_temperature=0 temperature=1.0000 has_top_p=0 top_p=1.0000 messages=4 stop=0
15:02:51.622 INF EncodeForContent:2080 | MiniCPM-V-4.6 pixel_values bytes=602112 min=2 max=250 (w=448 h=448 ps=14)
15:02:52.151 INF EncodeForContent:2124 | vision cache store: /tmp/axllm_media/media_863903737595189_1.png
15:02:52.225 INF SetKVCache:3134 | decode_grpid:0 prefill_grpid:1 history_cap:0 total_cap:128 symbolic_cap:1 precompute_len:0 input_num_token:358 prefer_symbolic_group:0
15:02:52.225 INF SetKVCache:3156 | current prefill_max_token_num:1280
15:02:52.242 INF SetKVCache:3172 | first run
15:02:52.258 INF Run:3376 | input token num : 358, prefill_split_num : 3
15:02:52.258 INF Run:3471 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=128
15:02:52.414 INF Run:3471 | prefill chunk p=1 history_len=128 grpid=3 kv_cache_num=256 input_tokens=128
15:02:52.587 INF Run:3471 | prefill chunk p=2 history_len=256 grpid=4 kv_cache_num=384 input_tokens=102
15:02:52.776 INF Run:3886 | ttft: 518.23 ms
这张图片展示了一个年轻的女性，她有着长长的银灰色头发，头发上装饰着一些花朵。她穿着一件浅灰色的比基尼，比基尼上有褶皱和蕾丝细节，显得非常优雅。她的面部表情平静，眼神直视前方，给人一种温柔和自信的感觉。背景是海滩，可以看到海浪轻轻拍打着沙滩，远处有绿色的树木，整个场景显得非常宁静和自然。女性的姿态轻松，似乎在享受海滩上的美好时光。

15:02:57.605 NTC Run:4298 | hit eos,decode avg 18.64 token/s
15:02:57.605 INF GetKVCache:3087 | precompute_len:449, remaining:831 (tracked)