inclusionAI / LLaDA2.0-mini

LLaDA2.0-mini

LLaDA2.0-mini is a diffusion language model featuring a 16BA1B Mixture-of-Experts (MoE) architecture. As an enhanced, instruction-tuned iteration of the LLaDA series, it is optimized for practical applications.

🚀 Performance Highlights

• Leading MoE Architecture : The open-source Mixture-of-Experts (MoE) diffusion large language model , pre-trained from scratch on approximately 20 trillion tokens .
• Efficient Inference : With 16 billion total parameters , only 1.4 billion are activated during inference. LLaDA2.0-mini significantly reduces computational costs while outperforming open-source dense models of similar scale.
• Impressive Performance on Code & Complex Reasoning : Excels in tasks such as code generation and advanced mathematical reasoning , demonstrating strong reasoning capabilities.
• Tool Use : Supports tool calling and achieves excellent performance in complex agent-based tasks.
• Open & Extensible : Fully open-source with commitment to transparency. We plan to release a leading inference framework in the future and continue investing in cutting-edge areas like diffusion LLMs (dLLM) to drive disruptive innovation.

🗺️ What's Next

• Supercharged Reasoning with LLaDA 2.0: LLaDA 2.0 series will be fine-tuned with Reinforcement Learning , unlocking a new level of sophisticated reasoning and problem-solving abilities.
• Tools for Innovators: The model was finetuned on the dFactory framework using Fully Sharded Data Parallel (FSDP2). We have begun open-sourcing dFactory and will continuously release our advanced post-training technologies. Whether you want to master the current model or build your own customized versions, you'll have the tools you need. Stay tuned for more updates!

📦 Model Variants

🔍 Model Overview

LLaDA2.0-mini has the following specifications:

• Type : Mixture-of-Experts (MoE) Diffusion Language Model
• Total Parameters (Non-Embedding) : 16B
• Number of Layers : 20
• Attention Heads : 16
• Context Length : 32,768 tokens
• Position Embedding : Rotary (RoPE)
• Vocabulary Size : 157,184

🤗 Hugging Face Transformers

Make sure you have transformers and its dependencies installed:

import torch
import torch.nn.functional as F
from transformers import AutoModelForCausalLM
from transformers import AutoTokenizer

model_path = "/path/to/LLaDA2.0-mini"
device = "cuda:0"
model = AutoModelForCausalLM.from_pretrained(
    model_path, trust_remote_code=True, device_map=device
)
model = model.to(torch.bfloat16)
model.eval()
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

prompt = "Why does Camus think that Sisyphus is happy?"
input_ids = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    add_generation_prompt=True,
    tokenize=True,
    return_tensors="pt",
)
generated_tokens = model.generate(
    inputs=input_ids,
    eos_early_stop=True,
    gen_length=512,
    block_length=32,
    steps=32,
    temperature=0.0,
)
generated_answer = tokenizer.decode(
    generated_tokens[0],
    skip_special_tokens=True,
)
print(generated_answer)

Best Practices

To achieve optimal performance, we recommend the following settings:

1. Sampling Parameters : We suggest using Temperature=0.0 , block_length=32 , and steps=32 . Using a higher temperature value may occasionally result in language mixing and a slight decrease in model performance.

2. Adequate Output Length : We recommend using an output length of 32768 tokens for most queries.

🌐 License

This project is licensed under the terms of the Apache License 2.0 .

🤝 Contact & Collaboration

For questions, collaborations, or feedback, please reach out via Hugging Face or open an issue in the repository .

👉 Join us in advancing open, efficient, and intelligent language models!