Pluto-AI-Labs / Atlas-Coder-0.5B

---

Model Description

Atlas-Coder-0.5B is a coding-specialized language model instruction-tuned from scratch on top of Qwen2.5-Coder-0.5B base (not instruct). Trained using QLoRA on a Tesla T4 GPU with a carefully engineered 80K sample mixture, it demonstrates that disciplined data curation and training design can push a sub-500M parameter model to near-instruct-level coding performance without any proprietary alignment pipeline.

This model is part of the Pluto AI research project by Siddharth N.R., following the Pluto-Genesis-0.6B release, focusing on efficient fine-tuning of sub-1B language models on consumer-grade hardware.

Research Goal: Prove that a sub-1B coding model fine-tuned on curated, decontaminated open-source data can match or exceed the coding performance of officially instruction-tuned variants of the same architecture — without RLHF, proprietary data, or large-scale compute.

---

⚠️ Benchmarks

⚠️ Note: This is an Infrastructure Case Study, not a SOTA Benchmark model.

Why is the score low?

This V1 model was fine-tuned on the Qwen2.5-Coder-0.5B-Base model using ChatML format. Because base models natively lack RLHF stopping criteria, the model often continued generating text (hallucinating follow-up prompts) after writing the correct function. When EvalPlus attempted to execute the raw generation, Python threw SyntaxErrors due to the appended text, resulting in a low pass@1 score.

---

Training Details

---

Training Data

Decontamination

All datasets were scanned using n-gram Jaccard similarity (8-gram, threshold 0.3) against the full HumanEval test set before training. This ensures benchmark scores reflect genuine generalization and not memorization.

---

Key Engineering Decisions

1. Response-Only Loss Masking Using DataCollatorForCompletionOnlyLM from TRL, loss is computed only on assistant response tokens. This prevents the model from wasting gradient steps learning to predict system prompts and user messages — the single highest-ROI change for HumanEval+ performance.

2. Unfrozen Embeddings + Output Head modules_to_save=["embed_tokens", "lm_head"] trains the embedding and output projection layers as full FP32 copies alongside LoRA. Critical when fine-tuning from a base (not instruct) model — the token distribution needs to shift significantly to learn the ChatML instruction format.

3. FP32 LoRA Cast on T4 PEFT 0.17 initializes LoRA matrices in BF16 by default. Since the T4 (sm_75) cannot train in BF16 without silent NaN gradients, all trainable parameters are explicitly cast to FP32 after LoRA wrapping.

4. Exec-Verified OSS Data as Primary Source 50K of 80K samples (62.5%) come from self-oss-instruct-sc2-exec-filter-50k — execution-verified, single-function Python completions derived from real open-source code. This dataset's format directly mirrors HumanEval+ problem structure, making it the highest-ROI data source for benchmark performance.

5. 3-Layer Checkpoint Recovery Training was designed to survive Kaggle's 12-hour session limit via a 3-layer resume system: local checkpoint scan → HuggingFace Hub download → fresh start. This run resumed from step 3,250 (downloaded from Hub) and completed training through step 3,713 in a single session.

---

Usage

Basic Inference

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "Siddh07ETH/Atlas-Coder-0.5B",
    torch_dtype=torch.float16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Siddh07ETH/Atlas-Coder-0.5B")

messages = [{"role": "user", "content": "Write a Python function to find the longest common subsequence of two strings."}]

text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=0.3,
        do_sample=True,
        top_p=0.9,
        repetition_penalty=1.1,
    )

response = tokenizer.decode(
    output[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True
)
print(response)

Low Memory Inference (4-bit)

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)
model = AutoModelForCausalLM.from_pretrained(
    "Siddh07ETH/Atlas-Coder-0.5B",
    quantization_config=quant_config,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Siddh07ETH/Atlas-Coder-0.5B")

GGUF (Ollama / LM Studio / llama.cpp)

GGUF quantizations for CPU inference are available at:

Siddh07ETH/Atlas-Coder-0.5B-GGUF

Runs at 40+ tokens/second on a laptop CPU using LM Studio, Ollama, or llama.cpp.

---

Recommended Generation Settings

---

Limitations

• Size: At ~494M parameters this model will make mistakes on complex multi-file engineering tasks. Always review generated code before running it.
• Context length: Trained on sequences up to 1024 tokens. Performance may degrade on prompts requiring longer context.
• Language bias: Optimized primarily for Python. Performance on other languages varies.
• Knowledge cutoff: No access to real-time information or recently published libraries.
• Research only: Not intended for production deployment without further evaluation and safety testing.

---

Comparison to Base Model

This model was fine-tuned from Qwen2.5-Coder-0.5B base , not instruct. The gap this training bridges:

Benchmark results will be published shortly via EvalPlus.

---

Related Models

---

Author

Siddharth N.R. (Siddhu) Final-year B.Tech — AI & Data Science Pluto AI Research

---

Citation

@misc{atlascoder2026,
  author    = {Siddharth N.R.},
  title     = {Atlas-Coder-0.5B: A QLoRA-Trained Sub-1B Coding Model from Base},
  year      = {2026},
  publisher = {HuggingFace},
  url       = {https://huggingface.co/Siddh07ETH/Atlas-Coder-0.5B}
}

---

License

Apache 2.0 — see LICENSE . Base model Qwen2.5-Coder-0.5B is also Apache 2.0.