fwerkor / MiniCPM5-2B-Diffusion-Base

MiniCPM5-2B-Diffusion-Base

MiniCPM5-2B-Diffusion-Base is a bidirectional masked-diffusion language-model backbone converted from openbmb/MiniCPM5-2B-Base . It is intended as a diffusion-native base model for CID and related research, rather than as an instruction-tuned chat model.

The checkpoint keeps the original Llama-compatible parameter layout, while the repository supplies a custom Transformers class that changes the forward pass to full-sequence bidirectional denoising. Loading it as a plain LlamaForCausalLM would be semantically incorrect.

Loading

Use Transformers with remote model code enabled:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "fwerkor/MiniCPM5-2B-Diffusion-Base"

tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to("cuda").eval()

The loaded class is CIDDiffusionForMaskedLM. Its forward pass is bidirectional, uses the checkpoint's dedicated mask token, and computes same-position masked denoising logits.

Diffusion generation

The custom class overrides model.generate so it does not fall back to autoregressive GenerationMixin semantics.

prompt = tokenizer(
    "The capital of France is",
    add_special_tokens=False,
    return_tensors="pt",
).input_ids.to("cuda")

output = model.generate(
    prompt,
    max_new_tokens=32,
    diffusion_steps=32,
    block_length=8,
)

print(tokenizer.decode(output[0], skip_special_tokens=True))

For direct masked-token denoising, use model.denoise(input_ids, steps=...).

Stage 0 conversion

The released checkpoint is the completed Stage 0 conversion run:

• Base: openbmb/MiniCPM5-2B-Base
• Objective: LLaDA-style absorbing masked diffusion
• Attention: bidirectional
• Sequence length: 2,048
• Optimizer steps: 5,087
• Global batch size: 96 sequences
• Tokens processed: 1,000,144,896
• Learning rate: 2e-5, constant
• Weight decay: 0.1
• Precision: BF16
• Mask-ratio sampling: uniform over [0.001, 1.0]

The streaming corpus mixture was:

Exact source revisions are recorded in diffusion_config.json.

Validation notes

The conversion substantially improves masked-token denoising over the untouched AR base. In a held-out smoke test used during release validation, the converted checkpoint reduced masked-token cross entropy from roughly 15.9/16.3/19.4 to 4.14/5.24/8.58 at 15%/50%/85% masking respectively.

This is a base checkpoint, not an instruction model. High-mask and free-form generation remain materially harder than low- and medium-mask reconstruction, and long unconstrained generations may repeat. Downstream CID training is expected to specialize the diffusion backbone further.

Paper and citation

This diffusion backbone is developed as part of Continuous Interaction Diffusion (CID) . See Continuous Interaction Diffusion: A Diffusion-Native Runtime for Asynchronous Tool-Augmented Reasoning ( arXiv:2608.10438 , DOI ).

If you use this model or the CID runtime, please cite:

@article{cao2026continuous,
  title   = {Continuous Interaction Diffusion: A Diffusion-Native Runtime for Asynchronous Tool-Augmented Reasoning},
  author  = {Cao, Yuhang},
  journal = {arXiv preprint arXiv:2608.10438},
  year    = {2026},
  doi     = {10.48550/arXiv.2608.10438},
  url     = {https://arxiv.org/abs/2608.10438}
}

Files and semantics

• config.json: Transformers config with the CID diffusion auto-map.
• modeling_cid_diffusion.py: self-contained bidirectional forward and diffusion sampler.
• diffusion_config.json: Stage 0 provenance and training metadata.
• model-*.safetensors: BF16 model weights.
• tokenizer.json / tokenizer_config.json: MiniCPM5 tokenizer plus the dedicated CID mask token.

The dedicated mask token is <|cid_mask|>.

License and attribution

This derivative checkpoint follows the Apache-2.0 license of openbmb/MiniCPM5-2B-Base. See the upstream model card for its full attribution, limitations, and citation information.