Akahsizrr / makeitwork-1

makeitwork-1

A 500M parameter decoder-only transformer (LLaMA-style architecture) fine-tuned for codebase information retrieval and tool-calling .

Model Details

• Architecture: Retriever500M (custom LLaMA-style decoder)
• Parameters: ~497M
• Hidden dim: 1,280
• Layers: 23
• Attention heads: 20
• FFN dim: 3,456 (SwiGLU)
• Positional encoding: RoPE
• Normalization: RMSNorm
• Tied embeddings: Yes
• Vocab size: 32,009 (with special tokens)

Capabilities

The model acts as a code search agent. Given a natural language query about a codebase, it:

1. Uses <|search|> to issue search queries
2. Analyzes <|result|> blocks from the retrieval system
3. Uses <|reasoning|> to think through findings
4. Returns curated <|evidence|> with relevant code facts
5. Ends with <|finish|>

Special Tokens

Evaluation Results

Evaluated on a held-out test set of 50 code search queries across Python, Rust, Go, C, and TypeScript codebases.

Training

• SFT Round 2 checkpoint
• Trained on a dataset of ~3,000 code search traces (v3 dataset)
• Traces generated from 20 open-source repositories
• Fine-tuned from the base Retriever500M model

Usage

import torch
import sys
sys.path.insert(0, ".")  # model.py in repo root

from model import ModelConfig, Retriever500M
from safetensors.torch import load_file
from tokenizers import Tokenizer

# Load config
import json
with open("config.json") as f:
    cfg = json.load(f)

config = ModelConfig(
    vocab_size=cfg["vocab_size"],
    d_model=cfg["d_model"],
    n_layers=cfg["n_layers"],
    n_heads=cfg["n_heads"],
    d_ff=cfg["d_ff"],
    max_seq_len=cfg["max_seq_len"],
    dropout=0.0,
    tie_embeddings=True,
)

model = Retriever500M(config)
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict)
model.eval()

tokenizer = Tokenizer.from_file("tokenizer_agent.json")

License

MIT