Jahirrrr / Klyra-64M

Klyra-64M

Klyra-64M is a compact language model family developed as an academic research project. The project was initiated and developed by students of Informatics Engineering at Politeknik Negeri Jakarta (State Polytechnic of Jakarta) , with a focus on efficient language modeling, staged post-training, reasoning, reinforcement learning, and model distillation at the sub-100M parameter scale.

The project explores how far a ~64M-parameter decoder-only language model can be pushed through staged pretraining, instruction tuning, reasoning-focused training, reinforcement learning, and specialist policy distillation.

Klyra is derived from the MiniMind-compatible architecture/runtime , with a custom training pipeline, data curriculum, post-training stages, evaluation workflow, and model checkpoints.

Current recommended checkpoint: Klyra-64M-Reasoning-Fundamentals
Experimental post-RL checkpoint: Klyra-64M-RL-OPD

---

Highlights

• ~64M trainable parameters
• 8 transformer layers
• 768 hidden size
• 8 attention heads / 4 KV heads
• ~6.4K Byte-Level BPE vocabulary
• 1024-token training context
• ~3B cumulative pretraining + midtraining tokens
• General instruction tuning
• Reasoning-fundamentals stage
• Specialist GRPO branches
• Online Policy Distillation (OPD)
• Evaluated on the same five English benchmark families used in the MiniMind-3 objective evaluation suite

Klyra is designed as a compact research model suitable for low-resource inference, controlled experimentation, and future edge or mobile deployment studies.

---

Model Architecture

The model uses tied token embedding / LM-head weights. Parameter counts may therefore differ slightly depending on whether a tool counts tied tensors once or separately.

---

Training Lineage

Random Initialization
        │
        ▼
Foundation Pretraining
~1B tokens
        │
        ▼
Midtraining
+~2B tokens
(~3B cumulative)
        │
        ▼
General Instruction SFT
        │
        ▼
Reasoning Fundamentals SFT
        │
        ├───────────────┬────────────────┬────────────────┐
        │               │                │                │
        ▼               ▼                ▼                ▼
Reasoning RL1      Instruction RL    General RL       Base student
        │
        ▼
Reasoning RL2
        │               │                │
        └───────────────┴────────────────┘
                        │
                        ▼
             Online Policy Distillation
                        │
                        ▼
                Klyra-64M-RL-OPD

---

Available Weights

The public Klyra-64M release currently consists of the following five checkpoints. These are the only model weights currently distributed as part of the public Klyra-64M release .

Project repository: Jahirrrr/Klyra-64M

Recommended checkpoint

For general use and downstream experimentation, the recommended checkpoint is:

Jahirrrr/Klyra-64M-Reasoning-Fundamentals

It currently provides the strongest overall balance between the five-task benchmark mean and manually evaluated reasoning behavior.

Experimental RL checkpoint

The final RL + OPD checkpoint is available at:

Jahirrrr/Klyra-64M-RL-OPD

This checkpoint is retained as an experimental research release. It broadly preserves aggregate benchmark capability, while its behavior differs from the recommended Reasoning-Fundamentals checkpoint on several manual reasoning and instruction-following tests.

Release note: Klyra-64M-Reasoning-RL1 , Klyra-64M-Reasoning-RL2 , Klyra-64M-IF-RL , and Klyra-64M-General-RL were intermediate specialist checkpoints used during the RL/OPD research pipeline. They are not currently distributed as public Klyra-64M weights .

---

Stage 1 — Foundation Pretraining

The model was initialized from random weights and pretrained on approximately 1B tokens .

Primary corpus:

• openbmb/Ultra-FineWeb-L1
• configuration: CC-MAIN-2025-30

Core settings:

Foundation validation result:

---

Stage 2 — Midtraining

Klyra was then continued for approximately 2B additional tokens , giving approximately 3B cumulative pretraining + midtraining tokens .

Training mixture:

Core settings:

Midtraining validation:

The midtraining validation set is not directly comparable with the foundation validation set.

---

Stage 3 — General Instruction SFT

Dataset:

• HuggingFaceTB/smol-smoltalk

After assistant-target validation and filtering:

Training:

• 2 epochs
• 1024 context
• assistant-only loss
• dynamic padding
• BF16
• peak LR approximately 5e-4
• 10% warmup

Results:

This produced:

Klyra-64M-Instruct

---

Stage 4 — Reasoning Fundamentals

The next stage used a conservative training strategy intended to strengthen fundamental reasoning while limiting degradation of previously acquired capabilities.

Instead of immediately pushing Klyra into difficult competition-style reasoning, the goal was to strengthen:

• arithmetic
• unit-rate reasoning
• short multi-step reasoning
• simple logical deduction
• basic science
• algorithm fundamentals

Training set:

30,000 total examples

18,000 reasoning examples
12,000 general-instruction replay

Evaluation set:

1,000 total examples

600 reasoning
400 general-instruction replay

Training:

The resulting checkpoint is:

Klyra-64M-Reasoning-Fundamentals

This remains the recommended Klyra-64M checkpoint .

---

Stage 5 — Specialist Reinforcement Learning

The Klyra reinforcement-learning experiment started from Klyra-64M-Reasoning-Fundamentals .

Three specialist branches were created.

Reasoning RL

Reasoning RL used a two-stage curriculum.

Primary datasets:

• openai/gsm8k
• ChilleD/SVAMP
• tasksource/ruletaker

Reasoning RL1

Approximate composition:

1,400 GSM8K short/easy
400 SVAMP
700 RuleTaker depth-1
-------------------------
2,500 prompts

Reasoning RL2

Approximate composition:

1,200 GSM8K medium
200 SVAMP
600 RuleTaker depth-2/depth-3
------------------------------
2,000 prompts

RL used a native GRPO-style implementation with:

• grouped generations
• group-relative advantages
• clipped policy ratio
• reference-policy KL regularization
• programmatic/verifiable rewards

---

Instruction-Following RL

The instruction-following branch used approximately 2,000 verified IFEval/RLVR-style prompts .

Source:

sungyub/ifeval-rlvr-verl

The verifier used classic IFEval-compatible rule checks.

The objective emphasized constraints such as:

• exact wording
• keyword inclusion/exclusion
• word/sentence counts
• JSON formatting
• capitalization rules
• punctuation rules
• section/title constraints
• response-format restrictions

Checkpoint:

Klyra-64M-IF-RL

---

General-Capability RL

General RL used train splits only from benchmark-family datasets:

Checkpoint:

Klyra-64M-General-RL

Test splits were not used for this RL stage.

---

Stage 6 — Online Policy Distillation

The OPD student starts again from:

Klyra-64M-Reasoning-Fundamentals

Teachers:

Reasoning teacher → Klyra-64M-Reasoning-RL2
IF teacher        → Klyra-64M-IF-RL
General teacher   → Klyra-64M-General-RL

Only teacher trajectories that passed objective verification were accepted for distillation.

Training:

Output:

Klyra-64M-RL-OPD

This checkpoint is designated as experimental and is not the recommended default checkpoint.

---

Benchmark Results

Evaluation protocol

Klyra is evaluated on the five English benchmark families used in the MiniMind-3 objective evaluation suite:

• ARC-Easy
• PIQA
• OpenBookQA
• HellaSwag
• Social-IQA

The Klyra rows below were produced with the same Klyra evaluation notebook, using conditional-likelihood scoring and the same task family across checkpoints. For the Klyra evaluation, the primary metrics are acc_norm for ARC-Easy, PIQA, OpenBookQA, and HellaSwag, and acc for Social-IQA.

External sub-100M results are included as a reference comparison , not as a universal leaderboard. Different public repositories may use different lm-evaluation-harness versions, model revisions, dtypes, tokenizer implementations, or prompt/chat-template settings. Therefore, cross-project differences should not be interpreted as controlled significance tests unless the models are rerun under one identical evaluation environment.

Klyra five-task benchmark

Klyra observations

• Klyra-64M-Reasoning-Fundamentals currently has the highest five-task Klyra mean at 36.92% .
• Klyra-64M-RL-OPD records the highest Klyra PIQA score at 59.36% and the highest Klyra Social-IQA score at 35.98% .
• The RL-OPD stage broadly preserves aggregate benchmark capability, but does not improve the overall five-task mean over Reasoning-Fundamentals .
• The MiniMind rows are author-reported reference results. The Klyra rows are produced by the Klyra benchmark notebook.

Broader sub-100M comparison

Many sub-100M models do not publish OpenBookQA and Social-IQA results. To include a substantially wider set of public models, the table below uses the three broadly overlapping zero-shot tasks:

ARC-Easy + PIQA + HellaSwag

Mean-3 is the unweighted arithmetic mean of those three scores. The table is sorted by Mean-3 . It is intended to provide context for Klyra rather than establish a universal SOTA ranking.

¹ The table uses the cRia Open SLM submission values for consistency with the broader comparison. The current cRia model card reports a later checkpoint with higher scores after additional training that includes one epoch over several benchmark-family training splits ; those later scores are therefore not substituted into this table.

---

Sub-100M Architecture and Training Comparison

The table below summarizes the architecture and training exposure of the principal sub-100M comparison models above. Pretraining / continued training refers to publicly disclosed token exposure for the cited checkpoint. A dash indicates that the corresponding value was not clearly disclosed in the public model card used for this comparison.

² tinctura-v1 is publicly described as a 96.2M-parameter, 18-layer variant of the cagliostro-v3 architecture/recipe trained to 75B tokens. The approximate width/head/vocabulary/context entries shown here correspond to that architecture family; the authoritative checkpoint configuration should be taken from the model repository when reproducing it.

Training-exposure perspective

Klyra's representation-learning exposure is comparatively small:

³ The cRia figure only reflects the disclosed base, released midtraining, and selected context-extension exposure. It excludes the later benchmark-family adaptation epoch because a single aggregate token count for that stage is not reported in the model card.

This comparison is useful for interpreting Klyra's current benchmark position: Klyra reaches its present scores after approximately 3B cumulative foundation and midtraining tokens, whereas many stronger sub-100M peers report tens of billions of training tokens.

External comparison sources

Public comparison values and architecture metadata were collected from the following model cards and public benchmark pages:

• AxiomicLabs Open SLM Leaderboard
• MiniMind-3
• tinctura-v1
• Rose-1.5-Medium
• Rose-Medium
• Rose-Mini
• Surjo-100m
• Surjo-50M
• cRia-LM-75M
• Kiyo-65M
• Supra-50M-Base
• BananaMind-2-Medium
• Veyra2-Apricot-50M-Base
• SmolLM2-70M
• Pythia-70M
• Stentor3-50M

The public sub-100M model ecosystem is larger than this table and changes over time. This section therefore represents a verified comparison set of relevant public peers with overlapping benchmark results , rather than an exhaustive catalog of every model below 100M parameters on the Hugging Face Hub.

---

Manual Reasoning Behavior

Reasoning-Fundamentals showed meaningful improvements on short fundamental reasoning tasks, including:

• integer addition with carry
• subtraction
• unit-rate problems
• basic syllogistic logic
• basic science facts
• binary-search explanation

Example:

Prompt:
What is 17 + 28?

Klyra:
<think>
In the ones place, 7 + 8 = 15; write 5 and carry 1.
In the tens place, 1 + 2 + carry 1 = 4.
So 17 + 28 = 45.
</think>
Final answer: 45.

---

Checkpoint Selection

Recommended general-purpose checkpoint

Use:

Jahirrrr/Klyra-64M-Reasoning-Fundamentals

This is currently the best-balanced public Klyra checkpoint.

General instruction baseline

Use:

Jahirrrr/Klyra-64M-Instruct

This checkpoint is useful as the general instruction-tuned baseline for comparisons with later reasoning and RL stages.

Experimental RL / OPD research

Use:

Jahirrrr/Klyra-64M-RL-OPD

This is the publicly released final checkpoint from the specialist-RL and Online Policy Distillation experiment. The individual specialist-teacher checkpoints are not currently released.

Continued-pretraining experiments

Use:

Jahirrrr/Klyra-64M-Midtrain

For experiments that require an earlier pretrained checkpoint, use:

Jahirrrr/Klyra-64M-Base

---

Tokenizer

Klyra keeps the existing MiniMind-style 6.4K Byte-Level BPE tokenizer .

Important tokens include chat/reasoning control tokens such as:

<|im_start|>
<|im_end|>
<think>
</think>
<tool_call>

The tokenizer was intentionally kept stable across the training lineage so checkpoints remain compatible.

---

Limitations

Klyra contains approximately 64M parameters and was exposed to approximately 3B cumulative pretraining and midtraining tokens prior to post-training.

Known limitations include:

• limited factual knowledge
• hallucination
• weak long-form coherence
• fragile multi-step reasoning
• operation-selection errors
• weak advanced mathematics
• limited coding ability
• repetition on longer generations
• sensitivity to prompt format
• weak exact instruction following in some checkpoints
• no guarantee of factual correctness
• 1024-token training context

The project is intended primarily as a study of capability development in compact language models and should not be interpreted as a substitute for substantially larger production-oriented models.

---

Training-Efficiency Context

Klyra has seen substantially fewer pretraining/midtraining tokens than many competitive sub-100M language models.

Klyra-64M
~3B cumulative pretraining + midtraining tokens
~64M parameters
≈47 representation-learning tokens per parameter

This relatively limited token budget motivates further continued-pretraining experiments.

---

Attribution

Klyra uses a MiniMind-compatible model architecture/runtime.

MiniMind:

https://github.com/jingyaogong/minimind

Klyra training, data curriculum, checkpoint lineage, reasoning stages, RL branches, evaluation workflow, and OPD experiments are separate work developed for the Klyra project.

Please preserve applicable upstream attribution and license notices when redistributing derived code or checkpoints.

---

License

Klyra is released under the Apache License 2.0 , subject to the licenses and terms of the upstream code and datasets used during training.

Individual training datasets may have their own licenses and usage requirements. Users are responsible for reviewing those terms before redistribution or commercial use.

---

Citation

If you use Klyra in research, you can cite the repository as:

@misc{klyra64m,
  title        = {Klyra-64M: A Compact Language Model with Staged Reasoning and Reinforcement Learning},
  author       = {Athallah, Zahir Hadi},
  year         = {2026},
  howpublished = {Hugging Face},
  note = {An academic research project developed by Informatics Engineering students at Politeknik Negeri Jakarta}
}

---

Model Family Summary

Klyra-64M-Base                         [Public]
    ↓
Klyra-64M-Midtrain                     [Public]
    ↓
Klyra-64M-Instruct                     [Public]
    ↓
Klyra-64M-Reasoning-Fundamentals       [Public · Recommended]
    │
    ├── Reasoning RL1                  [Intermediate · Not released]
    │       ↓
    │   Reasoning RL2                  [Intermediate · Not released]
    │
    ├── Instruction-Following RL       [Intermediate · Not released]
    │
    └── General-Capability RL          [Intermediate · Not released]
                 │
                 ▼
          Klyra-64M-RL-OPD             [Public · Experimental]

Main project repository: Jahirrrr/Klyra-64M
Recommended checkpoint: Jahirrrr/Klyra-64M-Reasoning-Fundamentals