Asilarkness / DiffuRefill-1B-base

DiffuRefill-1B (base)

A 1.08B-parameter masked-diffusion language model, pretrained from scratch for $0 on free, preemptible notebook GPUs.

• 200,000 steps, 92.8B tokens , 21 days (2026-09-10 → 2026-10-01)
• ~200 worker launches on free RTX PRO 6000 notebooks that are reclaimed without warning; the whole fleet was lost and rebuilt many times
• The only shared state was this Hub: workers train 150 steps alone, upload, and a restartable merger averages them (DiLoCo)
• 📄 Technical report: paper/DiffuRefill-1B-technical-report.pdf

This is a base model : no instruction tuning, no safety tuning. It writes fluent English and often gets facts wrong.

What it is

DiffuRefill is a bidirectional transformer trained with the masked-diffusion (MDLM) objective: random tokens are replaced by [MASK] and the model learns to fill them in from both sides. Instead of writing text strictly left to right, it starts from a row of masks and fills it in over several passes, and it can commit several tokens per pass.

Usage

# pip install torch transformers safetensors huggingface_hub
from huggingface_hub import hf_hub_download
import importlib.util, torch

path = hf_hub_download("Asilarkness/DiffuRefill-1B-base", "modeling_diffurefill.py")
spec = importlib.util.spec_from_file_location("modeling_diffurefill", path)
mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod)

model = mod.DiffuRefill.from_pretrained("Asilarkness/DiffuRefill-1B-base").cuda()

# one token per pass: slowest, most coherent
print(model.generate("The theory of relativity states that", max_new_tokens=64))
# 4 tokens per pass: ~4x fewer passes, some quality loss
print(model.generate("The theory of relativity states that", max_new_tokens=64, per_pass=4))

# scoring: log p(continuation | context), the scorer used for the benchmarks
print(model.loglikelihood("The capital of France is", " Paris."))   # -2.03
print(model.loglikelihood("The capital of France is", " Berlin."))  # -8.11

Benchmarks

Zero-shot, lm-evaluation-harness , every model run by us with the same harness and settings. Our model is plugged into the harness with a left-to-right chain-rule scorer ( eval/dlm_lmeval.py ). A Monte-Carlo ELBO estimator, as used by LLaDA, scored slightly lower (HellaSwag 40.8 vs 42.5 on 1000 items), so the scorer does not understate the model.

HellaSwag, ARC-c, PIQA, OBQA: length-normalised accuracy; the rest: accuracy. FLOPs ≈ 6·params·tokens.

Read this honestly. DiffuRefill-1B is clearly behind autoregressive models of its size. Every one of them used 3× (Pythia) to 280× (Qwen2.5-1.5B) more training compute, and masked diffusion is known to need much more compute than autoregression for the same likelihood (≈16× by Nie et al., 2025 ). It is relatively strong on SciQ (science facts, matching the fact-heavy anneal) and weakest on LAMBADA and WinoGrande (at chance). TinyLlama and OLMo-1B were also run but load incorrectly under the current transformers (e.g. OLMo LAMBADA 0%), so they are left out. All raw numbers are in eval/results.json .

Speed

Greedy generation of 256 tokens after a 32-token prompt, one RTX PRO 6000 (96 GB), bf16. Autoregressive models use Hugging Face generate() with a KV cache; DiffuRefill uses no cache and re-reads the whole row each pass.

At batch 1 (latency) diffusion wins: 1.9× at the same quality setting and 15× at 8 tokens per pass. At batch 32 (throughput) the KV cache wins unless diffusion commits 8 tokens per pass. More tokens per pass costs quality, and optimised AR servers such as vLLM are much faster than HF generate() .

How it was trained

Lessons (details in the report)

• ✅ Adopt the merged model by replacement . Keeping local drift made 60 of 63 merges worse.
• ✅ Halve the LR for the merge interval. Local drift fell 6× and a 1.3B-token plateau ended.
• ✅ Document masking and 10% short rows helped. Uniform weight averaging was the best final checkpoint.
• ❌ Outer Nesterov momentum, a mid-run switch to Muon , span/PMI masking , phrase tokens and cautious AdamW did not help this run.
• 🔬 On a small test model, writing a chain of thought first and then filling the answer in one parallel pass lifted a multi-step arithmetic task from 45% to 100%. That is the plan for the reasoning stage.

Limitations

This is one run, and interventions were chosen on small probes and a 38M test model. The fact probe overlaps with the Wikipedia-derived anneal data. The model is English only and has no safety tuning, so do not use it for anything that matters without further training.

Citation

@misc{diffurefill2026,
  title  = {DiffuRefill-1B: Pretraining a Masked Diffusion Language Model on Free, Preemptible Notebook GPUs},
  author = {Asilarkness},
  year   = {2026},
  url    = {https://huggingface.co/Asilarkness/DiffuRefill-1B-base}
}

Compute was provided free by molab notebooks. Engineering was done with Claude Code (Anthropic) as an assistant. The training run's working repository, with intermediate checkpoints and logs, is Asilarkness/DiffuRefill-1B .