litert-community / Laya-Multilingual-LiteRT

Laya Multilingual for LiteRT — Android GPU FP32

Run the multilingual checkpoint of convaiinnovations/laya on a phone GPU with LiteRT . Laya reads a text or a JSON state and answers questions you define at request time: pick one of several options, score on an ordinal scale, or give a yes/no probability. Each question is one forward pass. The validated Android configuration is LiteRT 2.2.0 with explicit GPU FP32 computation on a Samsung Galaxy S26 (SM-S942Q, Android 16): 51 ms of graph time per question at 256 tokens, 60 ms end to end in the sample app. On 201 English and Japanese question rows the answers match the official laya 0.3.4 fp32 CPU implementation: the same argmax on every choice and score question, and a maximum probability difference of 0.0014. Other Android GPU families have not been validated here.

Both panels are screenshots of the Android sample in android/ , taken on the Galaxy S26 with the GPU selected. The email is invented. The probabilities are the values the model returned. assets/demo.mp4 shows the same run.

Files and supported configuration

The recommended phone set is the S256 wfp16 graph, the act head, the token table, the tokenizer pair and the calibration file: 679,274,893 bytes.

The same checkpoint in the token-id form (table inside the graph, fp32 on the GPU) and the English checkpoint's fp32 graphs are in litert-community/laya-LiteRT .

wfp16 files store the 99 FULLY_CONNECTED weight tensors as float16 with a DEQUANTIZE to float32. Every activation and every other constant stays float32.

The main graph holds the mmBERT-base encoder, Laya's two typed head layers and the option scorer, applied at every position. The host does the rest: tokenize, build the prompt with one <mask> marker per option, look up the embedding rows, read the logits at the marker positions, and apply softmax. The act head is a separate small graph because its input depends on those host-side probabilities.

The graphs take embedding rows, not token ids. With the token table inside the graph, the float16-weight file does not compile on LiteRT 2.2.0 CompiledModel GPU: the GPU delegate rejects the EMBEDDING_LOOKUP that reads a DEQUANTIZE-fed table ( Empty quantization params ), and CompiledModel needs every operator on the GPU. The fp32 file with the table inside does compile (1.29 GB, about 7 s of compilation; litert-community/laya-LiteRT ships that form). With the lookup on the host, the wfp16 graph compiles as one GPU partition (1779 of 1779 operators) at 251 MB and 1.2 s. The float16 table reproduces the checkpoint's float32 table exactly (maximum difference 0 over all 196,608,000 values).

Use explicit FP32 GPU computation , as in the Kotlin block below. It is the only GPU precision validated here. INT8 files were not built and NPU execution was not evaluated. The English and typed-decisions checkpoints of the same repository are in litert-community/Laya-English-LiteRT , in the same graph form. Questions with more than 20 options are outside what was validated.

Minimal usage

Python — complete pipeline, desktop CPU

Needs numpy , transformers (tokenizer only) and ai-edge-litert . Neither torch nor the laya package is required. Put laya_host.py next to the downloaded files.

import json
from pathlib import Path
from laya_host import LayaHost

assets = Path(".")
questions = {
    "intent": {
        "type": "choice",
        "instructions": "What does the customer need?",
        "criteria": {"refund": "money returned", "help": "technical help"},
    }
}
with LayaHost(
    tokenizer_dir=assets,
    main_graph_path=assets / "laya_ml_s256_embeds_wfp16.tflite",
    act_graph_path=assets / "laya_ml_act_head_fp32.tflite",
    window=256,
    head_max_len=256,
    temperatures_json=assets / "laya_ml_calibration.json",
    embeddings_path=assets / "token_embeddings_fp16.bin",
) as host:
    result = host.predict("同じ支払いが二重に請求されました。返金をお願いします。", questions)
    print(json.dumps(result, ensure_ascii=False, indent=2))

assets is the directory with the downloaded files; tokenizer_dir needs tokenizer.json and tokenizer_config.json . Run from a directory that held only the seven files it names, this exact code printed:

"choice": "refund", "probabilities": {"refund": 0.9954, "help": 0.0046},
"confidence": 0.958, "action": {"act_probability": 1.0}, "usage": {"input_tokens": 37, ...}

Kotlin — Android GPU with explicit FP32

This is LayaMinimalUsage.kt from the sample, compiled into the app (one comment trimmed). It gathers the float16 rows, writes the float32 buffers, runs both graphs and decodes. LayaEngine loads the tokenizer, the calibration file and the memory-mapped token table. Build and install steps are in android/README.md .

import android.content.Context
import androidx.annotation.WorkerThread

object LayaMinimalUsage {
  /** Call off the main thread after installing the files listed in README.md. */
  @WorkerThread
  fun classify(context: Context): Map<String, Any?> {
    LayaEngine(context, LayaEngine.Storage.WFP16).use { engine ->
      val question =
        linkedMapOf<String, Any?>(
          "type" to "choice",
          "instructions" to "What does the customer need?",
          "criteria" to linkedMapOf("refund" to "money returned", "help" to "technical help"),
        )
      val backend = LayaEngine.Backend.GPU
      // CompiledModel.GpuOptions(precision = CompiledModel.GpuOptions.Precision.FP32).
      engine.initialize(backend)
      val row = engine.prepare("同じ支払いが二重に請求されました。返金をお願いします。", question)
      val raw = engine.runRaw(row, backend)
      check(raw.finite)
      return LayaDecoder.decode(raw.markerLogits, raw.actLogits, row.question, engine.calibration)
    }
  }
}

Host contract

HOST_CONTRACT.md is the complete specification. In short:

1. Build the sequence exactly as the upstream build_sequence does: <bos> "<type> question: <instructions>" <eos> <mask> option0 <mask> option1 … <eos> text <eos> . Each option keeps at most 48 tokens. The text is truncated on the right so the whole sequence fits in N. Remember the position of every <mask> .
2. Pad the ids on the right with id 0 to N. inputs_embeds [1,N,768] holds the token-table row of every position, padding included. attention_mask [1,N] is 1 for real tokens. qtype_onehot [1,3] selects choice, score or yes/no.
3. The main graph returns token_logits [1,N] and pooled_cls [1,768] . Take the logits at the marker positions, divide by the temperature for that question type and option count, and apply softmax. That is the answer distribution.
4. The act head takes pooled_cls and four features of the uncalibrated distribution (top probability, top-two margin, normalized entropy, option count / 255) and returns act_logits [1,2] .

The sample app ships three upstream question presets unchanged: email triage, support intent and moderation.

Measured quality and performance

Reference: the official laya 0.3.4 predict on fp32 CPU at checkpoint revision 1c5edc17 , with max_len set to the window. The inputs are 44 invented states (21 English, 21 Japanese, 2 mixed) with the upstream presets, the upstream quickstart schema and custom schemas of up to 20 options: 201 question rows, of which 81 are choice or score questions and 120 are yes/no. Probabilities are compared at temperature 1, including the yes/no probability and the act probability.

On the phone the Kotlin tokenizer and prompt builder produced the same token ids and marker positions as the Python reference on all 201 rows. Desktop CPU is ai-edge-litert 2.1.6 on macOS arm64 with 4 threads.

The per-question time covers both graphs, from writing the inputs to the end of output readback: the median of 200 rows after one cold call, debug build, screen on, no other workload, battery at 35–37 °C. The host embedding lookup adds a median of 14.6 ms. In the non-debuggable build of the sample, a five-question email-triage run took 301 ms and 303 ms in two fresh launches, and the app reached Ready 2.3 s after onCreate . That start-up is 1.26 s of GPU compilation, 0.76 s of tokenizer loading and a 0.27 s warm-up pass. These are samples from one device, not a benchmark across devices or thermal states.

Calibration

The upstream multilingual checkpoint ships with every temperature at 1. laya_ml_calibration.json holds temperatures fitted here, one per question type and option-count bucket, on 4,415 labeled English and Japanese examples from public datasets, at the 256-token window. A temperature never changes which option wins.

Fits use the same number of English and Japanese rows. A bucket keeps T=1 when a fitted value makes either language worse by more than 0.05 ECE. That is the case for 6–10 options. For 2 options the fit helps Japanese and costs English 0.03, so the pooled figure rises. This is a generic starting point: refit on your own data before you rely on the probabilities. Sources, licenses and label mappings are in licenses/CALIBRATION_DATASETS.md .

Limits

• The numbers above measure agreement with the upstream model, not task accuracy. The conversion reproduces the upstream answers, including the wrong ones. On five of the Japanese rows the upstream answer differs from the intended reading; one example is a veiled threat that scores 0.0 for threat . They are listed in HOST_CONTRACT.md .
• The act probability was 1.0 on all 201 rows, in the upstream model and in these graphs. Do not treat it as an escalation signal without your own evaluation.
• The S512 graphs were checked on desktop CPU only. The calibration was fitted at 256 tokens.
• One device was tested. The token table is memory-mapped and the app needs about 680 MB of files in its private storage.

Provenance, conversion and license

• Source: convaiinnovations/laya , revision 1c5edc17a7acd8701df6fc341c0d179f1c62c982 , subfolder multilingual/ (encoder jhu-clsp/mmBERT-base , 22 layers, hidden 768, vocabulary 256,000), loaded with laya 0.3.4 and transformers 5.17.0.
• Conversion: litert-torch 0.9.3 ( torch 2.12.1), fixed shapes, fp32. The math is unchanged: host-side embedding lookup, the question type as a one-hot matmul, padding and sliding-window masks as float constants, rotary tables baked per layer type, attention kept at rank 4, the two head layers written out explicitly, exact GELU, and the scorer applied at every position.
• Weight storage: ai-edge-quantizer 0.8.0 float16 FLOAT_CASTING on the FULLY_CONNECTED weights only ( wfp16 files).
• Verification: LiteRT CompiledModel Python API on desktop CPU, and the CompiledModel Kotlin API on the Galaxy S26 for CPU and GPU. Every gate is the same argmax plus an absolute probability difference. Correlation was never used as a gate.

License: the Laya checkpoint and code are Apache-2.0, and these converted files are released under the same license. mmBERT-base declares MIT. The host code ports logic from laya , transformers and tokenizers (Apache-2.0). License texts are in licenses/ , and attribution is in NOTICE . The calibration datasets keep their own terms; none of their text is included here.