AXERA-TECH / gemma-3-270m-it

Gemma 3 270M IT on AXERA NPU

Ready-to-run deployment package for google/gemma-3-270m-it on AX650 / NPU3.

• This release packages the bf16 AXERA NPU text runtime.
• Compiled with Pulsar2 llm_build ( prefill_len=128 , kv_cache_len=1023 ).
• Includes the tokenizer/config files required at runtime.
• Includes the compiled Gemma 3 text .axmodel files and the packed token-embedding weights.
• Includes the validated bin/axllm binary for AX650 / NPU3 board-side deployment.
• Supports text-only chat through the axllm runtime.

Supported Platform

• AX650 / NPU3

Performance

All measurements below were taken on AX650 / NPU3 with the packaged runtime files. TTFT stands for time to first token. The packaged prefill window is 128 tokens; prompts longer than that are split into multiple prefill chunks, which is reflected in the last row.

• The numbers above are measured on-device with the axllm run interactive flow, not theoretical.
• A prompt that exceeds the 128 -token prefill window is processed in multiple chunks, so its TTFT is higher than a single-chunk prompt (about one extra chunk of prefill). The long prompt above ( 170 tokens, two chunks) correctly extracted "next Friday" as the answer from the passage. Decode throughput is unaffected by prompt length.
• Decode throughput is the average reported by the runtime after the response reached eos ; the small variation reflects the differing response lengths.

Startup Runtime Footprint

The flash total breaks down as 337 MiB for the 19 .axmodel files, 320 MiB for model.embed_tokens.weight.bfloat16.bin , and 16 MiB for tokenizer.txt .

Package Layout

.
├── README.md
├── config.json
├── post_config.json
├── tokenizer.txt
├── model.embed_tokens.weight.bfloat16.bin
├── bin/
│   ├── axllm
│   └── axllm.version.json
├── gemma3_text_p128_l0_together.axmodel
├── ... (gemma3_text_p128_l1..l17_together.axmodel)
├── gemma3_text_post.axmodel
└── gemma_3_270m_it_tokenizer/
    ├── config.json
    ├── tokenizer.json
    ├── tokenizer.model
    ├── tokenizer_config.json
    ├── generation_config.json
    ├── special_tokens_map.json
    ├── added_tokens.json
    └── chat_template.jinja

This package uses a flat runtime layout: the compiled text runtime files, the packed embedding weights, and tokenizer.txt live at the repository root so axllm can load the model directory directly. The bin/ directory holds the validated AX650 axllm binary ( axllm.version.json records the source branch, commit, and checksum it was built from). The original Hugging Face tokenizer/config files are kept in the gemma_3_270m_it_tokenizer/ subdirectory for reference. Model weights ( safetensors ) are not shipped in this package.

Direct Inference with axllm

Download the Model Package

Download the release package from Hugging Face:

mkdir -p AXERA-TECH/gemma-3-270m-it
cd AXERA-TECH/gemma-3-270m-it
hf download AXERA-TECH/gemma-3-270m-it --local-dir .

Install axllm

This package already includes a validated bin/axllm binary for AX650 / NPU3, so no build step is required — just make it executable:

chmod +x ./bin/axllm

If you prefer to build or fetch axllm yourself, use one of the following instead.

Option 1: clone the repository and run the installer:

git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm
./install.sh

Option 2: install with a one-line command (default branch: axllm ):

curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash

Option 3: download the prebuilt binary from GitHub Actions CI:

If you do not have a local build environment, download the latest CI-generated axllm binary from GitHub Actions: https://github.com/AXERA-TECH/ax-llm/actions?query=branch%3Aaxllm Then run:

chmod +x axllm
sudo mv axllm /usr/bin/axllm

Run on the Board

The package root is already arranged for axllm , so no extra runtime path arguments are required. The commands below use the packaged bin/axllm binary.

Note: the command below assumes you run it from the parent directory of AXERA-TECH/gemma-3-270m-it . If you are already inside the package directory, use ./bin/axllm run . instead.

$ ./AXERA-TECH/gemma-3-270m-it/bin/axllm run AXERA-TECH/gemma-3-270m-it

# output log example:
09:13:02.036 INF Init:1658 | LLM init start
09:13:02.037 INF Init:1689 | attention config: layers=18 sliding=15 full=3 linear=0 sliding_window=512 ref_full_layer_idx=0
tokenizer_type = 3
huggingface tokenizer mode = space_replace_bpe
100% | ################################ |  21 /  21 [5.45s<5.45s, 3.86 count/s] embed_selector init ok
09:13:07.483 INF Init:1839 | max_token_len : 1023
09:13:07.483 INF Init:1842 | kv_cache_size : 256, kv_cache_num: 1023
09:13:07.483 INF init_groups_from_model:1205 | prefill_token_num : 128
09:13:07.483 INF init_groups_from_model:1430 | prefill_max_token_num : 640
09:13:07.483 INF Init:1941 | LLM init ok
Commands:
  /q, /exit  退出
  /reset     重置 kvcache
  /dd        删除一轮对话
  /pp        打印历史对话
Ctrl+C: 停止当前生成
----------------------------------------
prompt >> What is the capital of France?
09:33:11.826 INF Run:3048 | ttft: 66.24 ms
The capital of France is Paris.

09:33:12.197 NTC Run:3448 | hit eos,decode avg 19.03 token/s
prompt >>

Interactive CLI

axllm run opens an interactive prompt. Type a message after prompt >> and the model streams a reply until it hits eos . Additional validated examples on AX650 / NPU3:

prompt >> Who wrote Romeo and Juliet?
The play "Romeo and Juliet" was written by William Shakespeare.

prompt >> Write a haiku about autumn.
Leaves turn gold and red,
Crisp air whispers through the trees,
Sunlight paints the world in hues.

prompt >> List the first five prime numbers.
The first five prime numbers are 2, 3, 5, 7, and 11.

Use /reset to clear the KV cache, /pp to print the conversation history, and /q or /exit to quit.

Serve with axllm

axllm serve exposes the same model through an OpenAI-compatible HTTP API.

Note: the command below assumes you run it from the parent directory of AXERA-TECH/gemma-3-270m-it . If you are already inside the package directory, use ./bin/axllm serve . --port 8000 instead.

$ ./AXERA-TECH/gemma-3-270m-it/bin/axllm serve AXERA-TECH/gemma-3-270m-it --port 8000
# output log example:
Starting server on port 8000 with model 'AXERA-TECH/gemma-3-270m-it'...
API URLs:
  GET  http://127.0.0.1:8000/health
  GET  http://127.0.0.1:8000/v1/models
  POST http://127.0.0.1:8000/v1/chat/completions
OpenAI API Server starting on http://0.0.0.0:8000
Max concurrency: 1
Models: AXERA-TECH/gemma-3-270m-it

Check the service health and list the available models:

$ curl http://127.0.0.1:8000/health
{
  "concurrency": 0,
  "max_concurrency": 1,
  "status": "healthy"
}

$ curl http://127.0.0.1:8000/v1/models
{
  "data": [
    {
      "id": "AXERA-TECH/gemma-3-270m-it",
      "object": "model",
      "owned_by": "openai-api"
    }
  ],
  "object": "list"
}

Send a chat completion request:

$ curl http://127.0.0.1:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "AXERA-TECH/gemma-3-270m-it",
      "messages": [{"role": "user", "content": "What is the capital of France?"}],
      "max_tokens": 64
    }'

# Example response:
{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "The capital of France is Paris.\n",
        "role": "assistant"
      }
    }
  ],
  "model": "AXERA-TECH/gemma-3-270m-it",
  "object": "chat.completion"
}

If you want a browser UI for the OpenAI-compatible service, use AXERA-TECH/lite_webui . Set the OpenAI base URL to http://<board-ip>:8000 and the model name to AXERA-TECH/gemma-3-270m-it .

Conversion References

If you need the original model files or want to rebuild the deployment artifacts, start with:

• Original Hugging Face model: google/gemma-3-270m-it
• AXERA conversion and deployment workflow: AXERA-TECH/gemma-3-270m-it.axera

When rebuilding for AX650 / NPU3, compile with FLOAT_MATMUL_USE_CONV_EU=1 pulsar2 llm_build ... --chip AX650 . This build used --hidden_state_type bf16 --prefill_len 128 --kv_cache_len 1023 .

Discussion

• GitHub Issues
• QQ group: 139953715