arcee-ai / Trinity-Mini-FP8-Block

Trinity Mini FP8-Block

This repository contains the FP8 block-quantized weights of Trinity-Mini (FP8 weights and activations with per-block scaling).

Trinity Mini is an Arcee AI 26B MoE model with 3B active parameters. It is the medium-sized model in our new Trinity family, a series of open-weight models for enterprise and tinkerers alike.

This model is tuned for reasoning, but in testing, it uses a similar total token count to competitive instruction-tuned models.

Trinity Mini is trained on 10T tokens gathered and curated through a key partnership with Datology , building upon the excellent dataset we used on AFM-4.5B with additional math and code.

Training was performed on a cluster of 512 H200 GPUs powered by Prime Intellect using HSDP parallelism.

More details, including key architecture decisions, can be found on our blog here

Try it out now at chat.arcee.ai

Model Details

• Model Architecture: AfmoeForCausalLM
• Parameters: 26B, 3B active
• Experts: 128 total, 8 active, 1 shared
• Context length: 128k
• Training Tokens: 10T
• License: Apache 2.0
• Recommended settings:
  • temperature: 0.15
  • top_k: 50
  • top_p: 0.75
  • min_p: 0.06

Quantization Details

• Scheme: FP8 Block (FP8 weights and activations, per-block scaling with E8M0 scale format)
• Format: compressed-tensors
• Intended use: High-throughput FP8 deployment of Trinity-Mini with near-lossless quality, optimized for NVIDIA Hopper GPUs
• Supported backends: DeepGEMM , vLLM CUTLASS, Triton

Benchmarks

Running our model

• VLLM
• Transformers

VLLM

Supported in VLLM release 0.18.0+ with DeepGEMM FP8 MoE acceleration.

# pip
pip install "vllm>=0.18.0"

Serving the model with DeepGEMM enabled:

VLLM_USE_DEEP_GEMM=1 vllm serve arcee-ai/Trinity-Mini-FP8-Block \
  --trust-remote-code \
  --max-model-len 4096 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_r1 \
  --tool-call-parser hermes

Serving without DeepGEMM (falls back to CUTLASS/Triton):

vllm serve arcee-ai/Trinity-Mini-FP8-Block \
  --trust-remote-code \
  --max-model-len 4096 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_r1 \
  --tool-call-parser hermes

Transformers

Use the main transformers branch

git clone https://github.com/huggingface/transformers.git
cd transformers

# pip
pip install '.[torch]'

# uv
uv pip install '.[torch]'

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "arcee-ai/Trinity-Mini-FP8-Block"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

messages = [
    {"role": "user", "content": "Who are you?"},
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    input_ids,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.15,
    top_k=50,
    top_p=0.75,
    min_p=0.06
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

API

Trinity Mini is available today on openrouter:

https://openrouter.ai/arcee-ai/trinity-mini

curl -X POST "https://openrouter.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "arcee-ai/trinity-mini",
    "messages": [
      {
        "role": "user",
        "content": "What are some fun things to do in New York?"
      }
    ]
  }'

License

Trinity-Mini-FP8-Block is released under the Apache-2.0 license.