kai-os / gemma4-31b-Opus-4.6-reasoning

Gemma 4 31B Opus Reasoning Adapter v1

This is a private QLoRA adapter for google/gemma-4-31B-it , fine-tuned on a cleaned subset of Crownelius/Opus-4.6-Reasoning-2100x-formatted .

The goal of this run was simple: produce a Gemma 4 31B reasoning adapter trained only on Opus-style reasoning data, without mixing in unrelated instruction corpora or agent traces.

Base Model

• Base model: google/gemma-4-31B-it
• Adapter type: LoRA / QLoRA ( peft )
• Quantization: 4-bit NF4
• Precision: BF16 compute

Dataset

Source dataset:

• Crownelius/Opus-4.6-Reasoning-2100x-formatted

Local filtering applied before training:

• Removed duplicate user prompts
• Removed obviously bad prompt families and formatting noise
• Kept reasoning-style rows only

Final local dataset stats:

• Rows in source: 2160
• Rows kept: 2025
• Train rows: 1924
• Validation rows: 101
• Category mix: 1899 math, 126 code

Training Setup

• Max sequence length: 4096
• Epochs: 2
• Learning rate: 1e-4
• Per-device batch size: 1
• Gradient accumulation: 8
• Hardware: NVIDIA GH200

LoRA target modules were adapted for Gemma 4 wrapped linear layers:

• q_proj.linear
• k_proj.linear
• v_proj.linear
• o_proj.linear
• gate_proj.linear
• up_proj.linear
• down_proj.linear

Validation Metrics

Final metrics from the completed run:

• Eval loss: 3.6018
• Eval perplexity: 36.66
• Train runtime: 3723s
• Epochs completed: 2.0

Published Base-Model Reference Benchmarks

The table below is included for context and comes from Google's official Gemma 4 31B Instruct model card . These are published base-model reference scores for google/gemma-4-31B-it , not adapter-specific evaluation results for this repository.

Source:

• google/gemma-4-31B-it

Usage

This repository contains a PEFT adapter, not a fully merged standalone model.

Load it with:

• Base model: google/gemma-4-31B-it
• Adapter: this repository

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
import torch

base_id = "google/gemma-4-31B-it"
adapter_id = "kai-os/gemma4-opus-reasoning-adapter-v1"

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(base_id)
base = AutoModelForCausalLM.from_pretrained(
    base_id,
    device_map="auto",
    quantization_config=bnb,
    torch_dtype=torch.bfloat16,
)
model = PeftModel.from_pretrained(base, adapter_id)

Notes

• This is a reasoning-focused adapter, not a benchmark-optimized release.
• The benchmark table above is for the published base model, not this adapter.
• It is best treated as an experimental distilled reasoning adapter.

Acknowledgements

• Google for Gemma 4
• The Opus reasoning dataset authors and maintainers
• Hugging Face transformers , peft , and datasets