DedeProGames / Kiyo-230M-Preview

Kiyo-230M-Preview

Kiyo-230M-Preview is an intermediate training checkpoint of Kiyo-230M, a decoder-only language model being pretrained from scratch on FineWeb-Edu, DCLM-Baseline, FineMath, and Stack-v3-train. It is released as a preview: the training run is still in progress and this checkpoint has seen roughly one fifth of its planned token budget. See Preview Status below before using or comparing it.

The architecture scales the Kiyo-135M design up to ~230M parameters: a Llama-style decoder with grouped query attention, RMSNorm, SwiGLU MLPs, and tied input/output embeddings, following the SmolLM2 architecture family and reusing its 49k-token vocabulary. Compared to Kiyo-135M, it goes deeper and wider — 32 layers at hidden size 768, against 30 layers at 576 — while keeping the same tokenizer, the same data mixture, and the same training code.

Preview Status

This repository holds the weights at optimizer step 81,500 of a run targeting 200B tokens . Three things follow from that, and all three matter when reading the benchmark numbers:

The learning rate has not decayed yet. The run uses a warmup-stable-decay schedule whose decay phase covers only the final 10% of steps. This checkpoint sits deep in the stable phase, still at the peak LR of 1.2e-3. Models evaluated mid-plateau consistently score below what the same weights reach after annealing, so the numbers below understate the model rather than describe it.

It has seen fewer tokens than Kiyo-135M. Kiyo-135M was trained on the full 200B tokens. This preview has seen 42.7B. A parameter-count comparison between the two is not a fair comparison of the architectures — it is a comparison of a finished run against a fifth of one.

Weights are not final and will change. Do not treat this checkpoint as the Kiyo-230M release. It exists so the run can be inspected mid-flight; the final model will be published separately.

Model Details

Training Data

The Stack-v3 portion is sampled with per-language keep probabilities across 32 curated programming languages, giving each language an equal token quota rather than inheriting the corpus's natural skew.

Benchmarks

Self-reported result from the official BananaMind Base Bench 1.1 script, measured on CUDA in float32.

Figures for BananaMind-2-Pro, Rose-Pro, and GPT-2 are as self-reported on their own model cards, all against the same BananaMind Base Bench 1.1 suite. This preview sits below Kiyo-135M despite having 1.7× the parameters, which is the expected result at 21% of the token budget with the learning rate still at peak — see Preview Status .

Detailed Kiyo-230M-Preview result

* = passes 1.96σ vs. chance; n=50 per category

Every category clears the significance threshold, including Quantitative — the one category where Kiyo-135M does not.

By difficulty

Summary

Scores are self-evaluated and may vary with the benchmark revision, Transformers version, dtype, hardware, and generation settings. With 350 items, the 95% interval spans roughly ±5 accuracy points, so small Elo gaps between models should not be read as decisive.

Usage

pip install -U transformers safetensors torch

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "DedeProGames/Kiyo-230M-Preview"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
).cuda().eval()

prompt = "The meaning of life is "
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)

with torch.no_grad():
    output = model.generate(
        input_ids=input_ids,
        max_new_tokens=64,
        do_sample=False,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

This is an unfinished base model, not instruction-tuned — it continues text rather than following instructions, and its weights are an intermediate checkpoint rather than a release. Beyond the preview caveats above, it shares the limits of its scale: fluent and grammatical on language completion, competent on code, but weak on multi-step logic and context tracking, where it stays near chance on the harder items. The 2,048-token context window limits long-document use. It can generate incorrect facts and should not be used for high-stakes decisions without verification. Keep a finite generation limit to avoid repetition or drift on long outputs.

License

Apache 2.0