kai-os / Grug-12B

Grug 12B

Grug 12B is a compact-reasoning fine-tune of google/gemma-4-12B-it . It was trained to keep the useful information from a reasoning trace while making the trace shorter, more direct, and less polished.

This repository is published as merged Transformers/safetensors model weights. It was trained with QLoRA, then merged into the base model for release.

GGUF

A llama.cpp Q4_K_M GGUF release is available in the adjacent repo: kai-os/Grug-12B-GGUF .

What Changed

The training target is a terse internal-reasoning style: short high-density steps, fewer filler words, and explicit preservation of key constraints, branching decisions, invariants, edge cases, and final-answer checks.

The goal is lower reasoning-token usage relative to the base model while preserving answer quality. It is not meant to hide uncertainty or remove needed reasoning.

Training Data

The data pipeline started from a recent, filtered reasoning pool and converted verbose traces into compact traces before SFT packing.

Source gate:

• Run date: June 30, 2026.
• Default freshness cutoff: 45 days. Sources older than May 16, 2026 were rejected unless manually allowed.
• Allowed train licenses: MIT, Apache-2.0, CC-BY-4.0, CC0-1.0.
• Hard reject terms included OpenAI, ChatGPT, GPT-5, Claude, Anthropic, Opus, Sonnet, and Gemini.
• Soft-risk sources marked as synthetic/distill were manually reviewed or rejected depending on provenance and license.

Final verified source mix:

Row counts:

• Normalized recent reasoning pool: 8,680 rows.
• Selected verbose reasoning set: 6,144 rows.
• Compact raw transform output: 6,144 rows.
• Verified compact rows: 5,740 rows.
• Rejected compact rows: 404 rows.
• Packed SFT split: 5,166 train / 287 validation / 287 test.

The compact reasoning transform was generated with cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit served by vLLM. Rows were checked for compression ratio, answer preservation, and obvious loss of critical reasoning information before training.

Training Procedure

Training was completion-only SFT: prompt tokens were masked with -100 , and only the assistant completion was trained.

Core settings:

• Base model: google/gemma-4-12B-it .
• Method: QLoRA / PEFT LoRA, merged into full model weights for upload.
• Quantization during training: 4-bit NF4 with BF16 compute.
• Max sequence length: 6,144.
• LoRA rank: 16.
• LoRA alpha: 32.
• LoRA dropout: 0.05.
• Target modules: q_proj , k_proj , v_proj , o_proj , gate_proj , up_proj , down_proj .
• Batch size: 1.
• Gradient accumulation: 16.
• Learning rate: 8e-5.
• Max steps: 100.
• Eval steps: 50.
• Save steps: 50.
• Train runtime: about 35 minutes 20 seconds on one A100.
• Final eval loss: 0.8895.

No train or validation rows were skipped in the final run.

Local Evaluation

Small local EOS-only math proxy eval, no generation token cap:

This is a small proxy eval, not a broad benchmark. Treat it as a smoke test showing the intended token-efficiency direction, then run your own benchmark.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "kai-os/Grug-12B"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model.eval()

messages = [
    {"role": "user", "content": "If a shirt is $80 and goes 25% off, what is the sale price?"}
]
inputs = tokenizer.apply_chat_template(
    messages,
    return_tensors="pt",
    add_generation_prompt=True,
).to(model.device)

with torch.no_grad():
    output = model.generate(inputs, do_sample=False, max_new_tokens=512)

print(tokenizer.decode(output[0], skip_special_tokens=True))

For token-efficiency tests, compare against the base model with the same prompt, same decoding settings, and no artificial token cap unless your deployment requires one.

Limitations

• This is an experimental fine-tune.
• It may over-compress reasoning on tasks that need longer derivations.
• It inherits the base model's limitations and safety behavior.
• The reported eval is small and local.
• The dataset includes synthetic and distilled reasoning traces from the listed open datasets; review source licenses and provenance before using this in commercial or sensitive settings.

Acknowledgements

Thanks to Lambda , the inference provider, for compute credits that supported the dataset work, training, and evaluation.