Akahsizrr / K2-Horizon-7B-LoRA-reasoning

K2-Horizon-7B LoRA Adapter — Reasoning/Math/Tools SFT

Training Details

• Base model: IFM/K2-Horizon-7B (9.00B params, custom K2Horizon architecture)
• Method: LoRA SFT (Supervised Fine-Tuning)
• LoRA config: r=64, alpha=64, dropout=0.05, target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Trainable params: 174.6M (1.90% of total)
• Training data: 798 golden rows selected from r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation
  • 403 reasoning rows (Qwen3.8-max teacher)
  • 260 math rows (Qwen3.8-max teacher)
  • 135 tool-calling rows (GLM-5.2 + K3 teachers)
  • Selection criteria: format_score >= 9.0, verifier_passed = True, no bad quality flags
• Training config:
  • Epochs: 2
  • Batch size: 4 per device, gradient accumulation 4 (effective batch 16)
  • Learning rate: 2e-4, cosine schedule, 10 warmup steps
  • Max sequence length: 2048
  • BF16 precision
  • Gradient checkpointing enabled
• Hardware: 1x A100-80GB (Modal)
• Training time: 13.3 minutes
• Loss trajectory: 2.046 → 0.448

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained("IFM/K2-Horizon-7B", trust_remote_code=True, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "Akahsizrr/K2-Horizon-7B-LoRA-reasoning")
tokenizer = AutoTokenizer.from_pretrained("Akahsizrr/K2-Horizon-7B-LoRA-reasoning", trust_remote_code=True)