hotchpotch / saihai-v1-310m

saihai-v1-310m

日本語での紹介記事はこちら: 日本語 Decision モデル "saihai" (采配) と、日本語 Decision モデルベンチマークの公開

saihai (采配) is a Japanese Typed Decision Model . saihai-v1-310m is its small encoder version: given a state and named questions, it returns probabilities for Noul (yes / no), Choice (one of several candidates) and Score (a rubric level), without generating text.

It is fine-tuned from the Japanese reranker cl-nagoya/ruri-v3-reranker-310m using the shared-prefix encoder and three decision heads of Bekko System One : the instruction and state are encoded once and reused for every candidate. The model weights are licensed under Apache-2.0, matching the Ruri base model.

日本語: saihai (采配) は、はい/いいえ (Noul)・選択 (Choice)・段階評価 (Score) の判断を、 文章を生成せずに確率として返す日本語の Decision モデルです。saihai-v1-310m は、 日本語 reranker の ruri-v3-reranker-310m をベースにした小型の encoder モデルです。

Other saihai models: saihai-v1-8b (8B decoder, higher accuracy) and saihai-v1-8b-gguf (llama.cpp). This 310M model does not run in llama.cpp; use Transformers as shown below.

Quick start

Use Python 3.12+ and install the standalone runtime dependencies. The Bekko and saihai training packages are unnecessary for inference.

uv pip install 'torch>=2.10,<2.11' 'transformers==5.17.0' 'sentence-transformers==6.1.0' 'safetensors>=0.7' 'tqdm>=4.67'

from transformers import AutoModel

model = AutoModel.from_pretrained(
    "hotchpotch/saihai-v1-310m",
    trust_remote_code=True,
    device="cuda",  # Use "cpu" for CPU inference.
    attn_implementation="sdpa",
)
request = {
    "state": "昨日購入した商品を返品したいです。返金をお願いします。",
    "questions": {
        "refund": {
            "type": "noul",
            "instructions": "顧客は返金を求めているか？",
            "criteria": {
                "false": "顧客は返金を求めていない。",
                "true": "顧客は返金を求めている。",
            },
        },
        "department": {
            "type": "choice",
            "instructions": "問い合わせを担当部署に振り分ける。",
            "criteria": {
                "refunds": "返品・返金の対応",
                "delivery": "配送状況・配達の対応",
                "technical": "ログイン・システム障害の対応",
            },
        },
        "urgency": {
            "type": "score",
            "instructions": "問い合わせの緊急度を評価する。",
            "criteria": [
                "通常対応：一般的な問い合わせや依頼",
                "優先対応：利用できず困っている",
                "至急対応：不正利用や被害が進行している",
            ],
        },
    },
}
response = model.systemone(request, show_progress_bar=False)
answers = response["answers"]

print(f"返金要求あり：{answers['refund']['noul']:.2%}")

department = answers["department"]
department_names = {
    "refunds": "返品・返金",
    "delivery": "配送",
    "technical": "技術サポート",
}
selected = department["choice"]
print(f"担当部署：{department_names[selected]}（{department['probabilities'][selected]:.2%}）")

urgency = answers["urgency"]
probabilities = urgency["probabilities"]
print(
    f"緊急度：通常対応 {probabilities['0']:.2%}、"
    f"優先対応 {probabilities['1']:.2%}、"
    f"至急対応 {probabilities['2']:.2%}"
    f"（スコア：{urgency['score']:.3f}）"
)

Example output, verified with CPU / FP32 / SDPA. CUDA uses BF16 autocast, so numerical differences may occur:

返金要求あり：95.57%
担当部署：返品・返金（99.98%）
緊急度：通常対応 43.53%、優先対応 28.09%、至急対応 28.38%（スコア：0.848）

The urgency score is the probability-weighted average of rubric indices 0 , 1 , and 2 . Probabilities are model outputs, not guaranteed correctness rates. Predictions depend on the supplied state, instructions and criteria.

The loader supports one CPU or CUDA device and a single-device map such as device_map={"": "cuda:0"} . Unlike the 8B loader, device_map="auto" automatic sharding is not supported. FP32 weights are loaded by default.

Interface and input limits

systemone() accepts state (a JSON value) and 1–64 named questions , then returns {"answers": ...} . Question order and criterion IDs are preserved. It supports the following convenient subset of the Saihai 8B interface; this wrapper does not return token usage or confidence estimates.

For a numeric rubric, use items such as {"description": "優先対応", "value": 2} . score remains the expected zero-based rubric index; expected_value averages the supplied numeric values. Values must be distinct finite numbers. No numeric grade is inferred from text. Questions are evaluated independently; this encoder reuses matching tokenized instruction/state prefixes across candidate branches within a microbatch.

The native Bekko interface remains available as model.predict(input_object) : use state_json and a decisions list with authored candidate descriptions, without targets or provenance. Native Score returns numeric expectation as score , plus normalized_score , values and probabilities . Raw-text encode() is not a typed-decision interface.

The runtime allocates the backbone's positional capacity adaptively between query and candidates, lending unused budget between them. The default candidate cap is 3,800 tokens including special tokens. Overlong queries use balanced truncation and candidates truncate on the right. This differs from the 8B model's reject-on-overflow policy. Training used query/candidate caps of 4,096/2,048; configured longer limits do not establish longer-input quality. For the recorded branch caps, pass query_length=4096, document_length=2048 to systemone() or predict() . See training/release-manifest.json for the complete evaluation conditions.

Batching and GPU execution

Pass a list of independent requests to batch across cases. Results follow input order; a single request returns one response, and a list returns a response list.

responses = model.systemone(
    [request, request],  # Replace with your application requests.
    batch_size=128,
    token_budget=64000,
    show_progress_bar=False,
)
print(responses[0]["answers"]["department"]["choice"])

batch_size bounds the request/decision window. token_budget bounds approximate padded query-plus-candidate work per microbatch; it is not a GPU memory limit. Complete decisions stay together, and length bucketing reduces padding.

Load with device="cuda" and attn_implementation="auto" to prefer compatible FlashAttention 2 on CUDA capability 8.0+ and otherwise use SDPA. Explicit "flash_attention_2" fails if the GPU or optional library is unsupported; "sdpa" needs no external attention wheel. Inspect model.runtime.attn_implementation for the selected backend. Install a FlashAttention wheel matching Python, PyTorch and CUDA before requesting FA2. Select the intended device/backend when loading; moving a CPU-loaded model does not automatically reselect its attention backend. CPU inference uses FP32; CUDA runtime inference uses BF16 autocast. Backend and precision changes can change probabilities.

For repeated GPU requests, compilation is optional:

model.compile_inference(mode="reduce-overhead", dynamic=True)
for _ in range(8):
    model.systemone(request, show_progress_bar=False)
response = model.systemone(request, show_progress_bar=False)
model.disable_compile()

Warm up representative lengths and candidate counts before measuring latency. First calls include compilation and new shapes may recompile. Only tensor execution is compiled; rendering and typed response construction stay eager. No latency result from another Bekko checkpoint is claimed for this model.

Training and reproducibility

Base revision: bb46934ee9ed09f850b9fcff17501b3ef7ddb2b3 . Training Dataset: hotchpotch/saihai-v1-310m-dataset , revision c00c6f122a42f950a5e58582d69c34d1263250c3 .

The exact pool contains 972,118 judgments in 820,822 train rows : 840,947 judgments from Bekko v0's capped train, selected using alpha=.5 square-root allocation without replacement and complete related groups, plus 131,171 Japanese judgments (including 14,400 Japanese LLM-jp gen_gen cases). English GPT-6 Sol gen_gen data is excluded; existing Bekko synthetic sources are retained. Source-specific licenses, notices and row lineage remain in the Dataset. The materialized pool is trained uniformly; do not apply alpha or a 10% fraction again.

Seed 42, batch size 512, one complete pass (1,899 updates), encoder / head learning rate 1e-5 / 2e-4, cosine schedule, 10% warmup, weight decay 0.01, full encoder fine-tuning without LoRA, gradient checkpointing, FlashAttention 2 with BF16, shared-prefix packing, query / candidate caps of 4,096 / 2,048 tokens, and both prefix orders sampled equally. The original validation pool has 14,416 rows / 21,884 judgments; a fixed balanced 120-case subset was monitored in both prefix orders every 250 updates.

Training guide: Training Saihai v1 310M in Bekko System One . The pinned recipe is configs/saihai-v1-310m.yaml . The original experiment used a development working tree based on trainer commit 30915a02ed80bc9d5691c3959eb43e30ae7be9ea ; its frozen source, dependency lock and hashes remain in this repository's history at revision 9068d16 . training/release-manifest.json records the training conditions and file hashes. The source checkpoint is included under checkpoint/ ; root weights are the independently loadable inference export.

Evaluation

Results on S1MB (System One Mosaic Benchmark). Scores are baseline-adjusted 0–100 values, not raw accuracy. Task Avg is the equal-weight average of the Noul, Choice and Score means; "general" columns average only the synthetic generalization tasks.

Compare with other decision models on the S1MB leaderboard . Both scopes completed with zero failed judgments, using CUDA BF16 autocast and the runtime's adaptive input budget (query 8,190 / candidate 3,800 tokens) with the instruction_state prefix order. The model is strongest on Choice tasks such as JCommonsenseQA, KUCI and JNLI; knowledge- and math-heavy synthetic tasks remain difficult, and Japanese General Score is at or below its reference baseline.

Training includes the train splits of seven Japanese source datasets evaluated by S1MB japanese-v1 (CommonsenseMoralJA, JCommonsenseQA, JNLI, JSTS, KUCI, Mintaka and JaNLI) and English source families from Bekko System One v0. Evaluation examples were excluded, but scores on those benchmarks are likely higher than on unseen tasks. These results do not establish unseen-task generalization or absence of pretraining exposure. Measurements taken at training time with the recorded 4,096 / 2,048 caps are in training/release-manifest.json .

Attribution and usage terms

The model weights, including the native checkpoint and inference export, are licensed under Apache-2.0 ; see LICENSE .

Base model: cl-nagoya/ruri-v3-reranker-310m , by the Ruri authors, licensed Apache-2.0. The original base license and source attribution are preserved in BASE-MODEL-LICENSE.txt . This model is a typed-decision finetune; it is not the original Ruri reranking release.

Training data has heterogeneous source licenses, including Japanese MIT / CC BY / CC BY-SA components and synthetic generation conditions. See the training Dataset's SOURCES.md. The model license does not replace the individual training-data licenses or generation-service terms. Training data is not relicensed under Apache-2.0; consult the Dataset source records for its terms.