Asilarkness / Budgie-500m

Budgie-500M

Budgie-500M is the strongest retained TestGeniy checkpoint: a 488M-parameter English causal language model for compact mathematical reasoning, formal logic, and dialogue with a 4,096-token context.

The repository root contains only final inference artifacts. There are no adapters, optimizer states, training rows, candidate folders, or obsolete checkpoints.

Controlled comparison

Both models used identical fixed examples and prompts with greedy decoding. Qwen used its native chat template with enable_thinking=False; Budgie used its native user/assistant template. This is a small controlled regression suite, not a full official leaderboard run.

Exact splits, revisions, seed, token budgets, counts, and raw percentages are recorded in benchmark.json .

Change from the previous root release

The retained candidate improves the controlled composite from 26.37% to 27.60% . The main gain is MATH-500; GSM8K and HelpSteer2 slightly regress, so the table is reported without cherry-picking.

Post-training

The starting checkpoint was Asilarkness/testgeniy at revision b85ebd6ed44c922a9fddcba79dd7f390be8d2c0c, path candidates/best-scaled-combo-a080.

The retained continuation used full-parameter BF16 SFT for 80 optimizer updates with four microbatches per update, AdamW (lr=2e-6, betas=(0.9, 0.95), weight_decay=0.1), cosine decay with 10% warmup, and gradient clipping at 1.0. The final release applies 50% of the learned task vector to the starting checkpoint to control forgetting.

Continuation data mixed externally sourced mathematics and reasoning from NuminaMath-CoT and Bespoke-Stratos, formal deduction from ProofWriter, and dialogue replay from SmolTalk. Held-out final and development questions were excluded by normalized exact matching and shared 12-token-window filtering. No benchmark test row was used for training.

Architecture

• 487,800,064 parameters
• 24 layers, hidden size 1,280
• 10 query heads, 2 KV heads, head dimension 128
• SwiGLU intermediate size 3,584 and tied embeddings
• 4,096-token context, RoPE theta 500,000
• hybrid attention with full NoPE attention every fourth layer

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Asilarkness/Budgie-500m"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).to("cuda").eval()
messages = [{"role": "user", "content": "What is 17 * 24? Explain briefly."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=160, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

trust_remote_code=True is required for the custom hybrid RoPE/NoPE architecture and digit-aware tokenizer.

Limitations

Budgie is experimental and remains behind Qwen3-0.6B on the controlled composite. It can produce incorrect, truncated, or repetitive reasoning. The reported benchmark samples are intentionally small and have wide uncertainty.

Provenance

• Base revision: b85ebd6ed44c922a9fddcba79dd7f390be8d2c0c
• Base path: candidates/best-scaled-combo-a080
• Released checkpoint: synth_sft_lr2e6_u80_a050
• Context length: 4,096 tokens