amd / Instella-3B

Instella✨: Fully Open Language Models with Stellar Performance

AMD is excited to announce Instella, a family of fully open state-of-the-art 3-billion-parameter language models (LMs) trained from scratch on AMD Instinct™ MI300X GPUs. Instella models outperform existing fully open models of similar sizes and achieve competitive performance compared to state-of-the-art open-weight models such as Llama-3.2-3B, Gemma-2-2B, and Qwen-2.5-3B, including their instruction-tuned counterparts.

By training Instella from scratch on Instinct MI300X GPUs, we highlight our hardware’s capability and scalability in handling demanding large-scale AI training workloads, offering a viable alternative in the AI hardware landscape. In line with the AMD commitment to open source, we are releasing all artifacts related to Instella models here , including the model weights, detailed training configurations, datasets, and code, enabling the AI community to collaborate, replicate, and innovate, thereby accelerating progress.

Takeaways

• Announcing Instella , a series of 3 billion parameter language models developed by AMD, trained from scratch on 128 Instinct MI300X GPUs.
• Instella models significantly outperform existing fully open LMs (Figure 1) of comparable size, as well as bridge the gap between fully open and open weight models by achieving competitive performance compared state-of-the-art open weight models and their instruction-tuned counterparts.
• Fully open and accessible: Fully open-source release of model weights, training hyperparameters, datasets, and code , fostering innovation and collaboration within the AI community.
• Supported by the AMD ROCm software stack, Instella employs efficient training techniques such as FlashAttention-2, Torch Compile, and Fully Sharded Data Parallelism (FSDP) with hybrid sharding to scale model training over a large cluster.

Instella Models

In this release, we introduce the following Instella models:

The Instella models are text-only, autoregressive transformer-based LMs having 3 billion parameters. Architecture-wise, Instella is packed with 36 decoder layers, each having 32 attention heads. These models support a sequence length of up to 4,096 tokens and have a vocabulary size of ~50,000 tokens using the OLMo tokenizer. During both pre-training and fine-tuning, we utilized FlashAttention-2, Torch Compile, and bfloat16 mixed-precision training to reduce memory usage, leading to computational speedups and optimal resource utilization. To balance inter-node memory efficiency and intra-node communication overhead within our cluster, we employed fully sharded data parallelism (FSDP) with hybrid sharding, with model parameters, gradients, and optimizer states sharded within a node and replicated across the nodes.

Our training pipeline is based on the open-sourced OLMo codebase, adapted, and optimized for our hardware and model architecture. For pre-training we used a total of 128 Instinct MI300X GPUs distributed across 16 nodes with each node having 8x Instinct MI300X GPUs. We evaluated our models and baselines using standard tasks from OLMES , FastChat MT-Bench , and Alpaca . For more details about the architecture, training pipeline/hyperparameters and evaluation results, please refer to our Blog , Hugging Face model card and Github repository .

Training Pipeline

The training of the Instella models comprised of four stages, where each stage incrementally enhanced the model’s capabilities from fundamental natural language understanding to instruction following and alignment towards human preferences.

Model Summary

Hyparparameter

Getting Started

Installation

First, install PyTorch according to the instructions specific to your operating system. For AMD GPUs, you can also start with a rocm/pytorch docker.

To install from source (recommended for training/fine-tuning) run:

git clone https://github.com/AMD-AIG-AIMA/Instella.git
cd Instella
# install Flash-Attention on MI300X
GPU_ARCH=gfx942 MAX_JOBS=$(nproc) pip install git+https://github.com/Dao-AILab/flash-attention.git -v
# install other dependencies
pip install -e .[all]

Example Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "amd/Instella-3B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto", trust_remote_code=True)

prompt = [{"role": "user", "content": "What are the benefits of open-source AI research?"}]
inputs = tokenizer.apply_chat_template(
    prompt,
    add_generation_prompt=True,
    return_tensors='pt'
)

tokens = model.generate(
    inputs.to(model.device),
    max_new_tokens=1024,
    temperature=0.8,
    do_sample=True
)

print(tokenizer.decode(tokens[0], skip_special_tokens=False))

Chat in TRL

You can also use the TRL CLI to chat with the model from the terminal:

pip install trl
trl chat --model_name_or_path amd/Instella-3B-Instruct --trust_remote_code --max_new_tokens 1024

# <root>:
# which is bigger 9.8 or 9.11?

# <amd/Instella-3B-Instruct>:
# 9.8 is bigger than 9.11. The difference between the two numbers is 0.69 (9.8 - 9.11 = 0.69), which indicates that 9.8 is 0.69 units larger than 9.11.  

Results

Pre-training

• Both Instella-3B-Stage1 & Instella-3B models outperform all the other fully open models over all the benchmarks individually (except PIQA). Our final pre-trained checkpoint Instella-3B outperforms the existing top performant fully open pre-trained models by a lead of ⬆️8.08% on average , with significant improvements in ARC Challenge [+8.02%], ARC Easy [+3.51%], Winnograde [+4.7%], OpenBookQA [+3.88%], MMLU [+13.12%] and ️GSM8K [+48.98%] .
• Second stage pre-training elevated the overall average performance relative to stage-1 by ⬆️5.26% , substantially narrowing the performance gap between Instella-3B model vs the closed-source models, and outperforming Llama-3.2-3B by ⬆️4.08% on average ( +5.69% [ARC Challenge], +5.61% [ARC Easy], and +29.72% [GSM8k] ), Gemma-2-2B by ⬆️7.25% on average ( +13.38% [ARC Challenge], +11.23% [ARC Easy], +4.5% [Hellaswag], +7.6% [OpenBookQA], +5.03% [MMLU], and +32.45% [GSM8k] ), and is competitive with Qwen-2.5-3B on the majority of the benchmarks.
• The multi-stage pre-training with diverse and high-quality data mix significantly enhanced Instella-3B’s capabilities, establishing it as a competitive and open alternative in the landscape of comparable size language models.

Instruction-tuning Results

• Instella-3B-Instruct model consistently outperforms other fully open models across all evaluated benchmarks with a significant average score lead of ⬆️ 14.37% w.r.t the next top performing fully open instruction-tuned models. With substantial margins across all the chat benchmarks ( +13% [MMLU], 7.57% [TruthfulQA], 7.43% [BBH], +4.46% [GPQA], +37.15 [IFEval], 10.08% [Alpaca 2], and 1.2% [MT-Bench] ).
• Instella-3B-Instruct narrows the performance gap with leading open-weight models. Instella-3B-Instruct performs on par with or slightly surpasses existing state-of-the-art open weight instruction-tuned models such as Llama-3.2-3B-Instruct ( +5.24% [TruthfulQA], 0.45% [GPQA], and +0.1% [MT-Bench] ), and Qwen2.5-3B-Instruct ( +2.01% [GPQA] and +8.87% [IFEval] ), while significantly outperforming Gemma-2-2B-Instruct with an average score lead of ⬆️5.83% ( +0.55% [MMLU], +3.79 [BBH], +4.91 [GPQA], +20.47 [GSM8k], +19.98 [Minerva MATH], and +15.17% [IFEval] ).
• Overall, Instella-3B-Instruct excels in instruction following tasks and multi-turn QA tasks like TruthfulQA, GPQA, IFEval and MT-Bench , while being highly competitive compared to existing state-of-the-art open weight models on other knowledge recall and math benchmarks, while being trained on significantly fewer training tokens.

Training Data

Further information concerning the training datasets, including applicable licensing terms and use restrictions, may be located at the linked source location.

Conclusion

The release of the Instella family of models represents a significant stride in advancing open-source AI and demonstrating the capabilities of AMD hardware in large-scale language model training. The 3 billion parameter models from Instella family significantly outperform present fully open comparable size models in key benchmarks while also being competitive to comparable open-weight models, which we attribute to the high-quality data-mix selection, multi-stage training pipeline, and the use of high-performance Instinct MI300X GPUs for large scale training.

By fully open sourcing the Instella models, including weights, training configurations, datasets, and code, we aim to foster innovation and collaboration within the AI community. We believe that transparency, reproducibility and accessibility are key drivers of progress in AI research and development. We invite developers, researchers, and AI enthusiasts to explore Instella, contribute to its ongoing improvement, and join us in pushing the boundaries of what is possible with language models.

We will continue enhancing the models across multiple dimensions, including context length, reasoning ability, and multimodal capabilities. Additionally, we will scale up both the model and dataset while exploring diverse architectural approaches. Keep your eyes peeled for more exciting blogs on the Instella LMs family, its features and capabilities!

Additional Resources

Hugging Face Model Cards

• Pre-trained models:
  • Instella-3B-Stage1: amd/Instella-3B-Stage1 , First stage pre-training checkpoint.
  • Instella-3B: amd/Instella-3B , Final pre-training checkpoint.
• Instruction-tuned models:
  • Instella-3B-SFT: amd/Instella-3B-SFT , Supervised fine-tuned checkpoint.
  • Instella-3B-Instruct: amd/Instella-3B-Instruct , Final Instruction-tuned checkpoint.

Datasets

Second stage pre-training GSM8k synthetic dataset: amd/Instella-GSM8K-synthetic

• The dataset consists of two splits: train and train_119K .
• For Instella-3B model second stage pre-training we used the train_119K split, which is a subset of the larger train split.

Code

• Github: https://github.com/AMD-AIG-AIMA/Instella

Please refer to the following blogs to get started with using these techniques on AMD GPUs:

• PyTorch Fully Sharded Data Parallel (FSDP) on AMD GPUs with ROCm™
• Accelerating Large Language Models with Flash Attention on AMD GPUs
• Accelerate PyTorch Models using torch.compile on AMD GPUs with ROCm™
• Introducing the First AMD 1B Language Models: AMD OLMo

Bias, Risks, and Limitations

• The models are being released for research purposes only and are not intended for use cases that require high levels of factuality, safety-critical situations, health, or medical applications, generating false information, facilitating toxic conversations.
• Model checkpoints are made accessible without any safety promises. It is crucial for users to conduct comprehensive evaluations and implement safety filtering mechanisms as per their respective use cases.
• It may be possible to prompt the model to generate content that may be factually inaccurate, harmful, violent, toxic, biased, or otherwise objectionable. Such content may also get generated by prompts that did not intend to produce output as such. Users are thus requested to be aware of this and exercise caution and responsible thinking when using the model.
• Multi-lingual abilities of the models have not been tested and thus may misunderstand and generate erroneous responses across different languages.

License

• The Instella-3B models are licensed for academic and research purposes under a ReasearchRAIL license.
• The amd/Instella-GSM8K-synthetic dataset used in second stage pre-training is built with Qwen2.5-72B-Instruct, and is licensed for academic and research purposes under a ReasearchRAIL license. Refer to the LICENSE and NOTICES in the amd/Instella-GSM8K-synthetic dataset card files for more information.
• Refer to the LICENSE and NOTICES files for more information.

Citations

Feel free to cite our Instella-3B models:

@misc{Instella,
    title = {Instella: Fully Open Language Models with Stellar Performance},
    url = {https://huggingface.co/amd/Instella-3B},
    author = {Jiang Liu, Jialian Wu, Xiaodong Yu, Prakamya Mishra, Sudhanshu Ranjan, Zicheng Liu, Chaitanya Manem, Yusheng Su, Pratik Prabhanjan Brahma, Gowtham Ramesh, Ximeng Sun, Ze Wang, Emad Barsoum},
    month = {March},
    year = {2025}
}