Total runs:
26.3K
Run Growth:
25.0K
Growth Rate:
97.36%
Sponsored by
APIMart
arcee-ai / Trinity-Large-TrueBase
Model's Last Updated: April 01 2026
text-generation
Introduction of Trinity-Large-TrueBase
Model Details of Trinity-Large-TrueBase
Trinity-Large-TrueBase
Introduction
Trinity-Large-TrueBase is a base pretraining checkpoint from Arcee AI's Trinity Large training run. It is a 398B-parameter sparse Mixture-of-Experts (MoE) model with approximately 13B active parameters per token. The checkpoint was captured after 10 trillion tokens of pretraining, prior to learning-rate annealing and before any instruction tuning or reinforcement learning.
This checkpoint is intended for research, probing, ablation studies, and downstream fine-tuning and comes without any pre-baked alignment, instruction formatting, or preference optimization.
More details on the training of Trinity Large are available in the technical report .
Model Variants
The Trinity Large family consists of three checkpoints from the same training run:
- Trinity-Large-TrueBase (this release): 10T-token pre-anneal checkpoint with no instruction data
- Trinity-Large-Base : Full 17T-token pretrained foundation model with mid-training anneals
- Trinity-Large-Preview : Lightly post-trained, chat-ready model undergoing active RL
Architecture
Trinity-Large-TrueBase uses a sparse MoE configuration designed to maximize efficiency while maintaining large-scale capacity.
| Hyperparameter | Value |
|---|---|
| Total parameters | ~398B |
| Active parameters per token | ~13B |
| Experts | 256 |
| Active experts | 4 |
| Routing strategy | 4-of-256 (1.56% sparsity) |
| Dense layers | 6 |
| Pretraining context length | 8,192 |
| Architecture | Sparse MoE (AfmoeForCausalLM) |
Note: Extended context support (e.g., 512k) was introduced after this checkpoint and is not available in TrueBase.
Benchmark Results
| Benchmark | N-shot | Metric | Score | Stderr |
|---|---|---|---|---|
| arc_challenge_0shot | 0 | acc_norm,none | 0.6237 | ±0.0142 |
| bbh_fewshot | 3 | exact_match,remove_whitespace | 0.5784 | ±0.0054 |
| gpqa_diamond_5shot | 5 | acc_norm,none | 0.4091 | ±0.0350 |
| gpqa_diamond_generative_5shot | 5 | exact_match,flexible-extract | 0.3788 | ±0.0346 |
| gsm8k_8shot | 8 | exact_match,flexible-extract | 0.8036 | ±0.0109 |
| gsm8k_cot | 8 | exact_match,flexible-extract | 0.8044 | ±0.0109 |
| hellaswag_5shot | 5 | acc_norm,none | 0.8813 | ±0.0032 |
| humaneval_plus | 0 | pass@1,create_test | 0.5183 | ±0.0391 |
| leaderboard_math_hard | 4 | exact_match,none | 0.2696 | ±0.0113 |
| mbpp_plus | 3 | pass_at_1,none | 0.8095 | ±0.0202 |
| minerva_math500 | 4 | math_verify,none | 0.4820 | ±0.0224 |
| mmlu_5shot | 5 | acc,none | 0.7845 | ±0.0033 |
| mmlu_generative_5shot | 5 | exact_match,get_response | 0.7848 | ±0.0033 |
| mmlu_pro | 5 | exact_match,custom-extract | 0.5160 | ±0.0044 |
| triviaqa_5shot | 5 | exact_match,remove_whitespace | 0.8096 | ±0.0029 |
| winogrande_5shot | 5 | acc,none | 0.8145 | ±0.0109 |
Training Configuration
Pretraining
- Training tokens: 10 trillion
- Checkpoint type: Pre-anneal
- Instruction data: None
- RLHF or post-training: None
This checkpoint branches from the main Trinity Large run at the 10T-token mark, prior to learning-rate decay or post-training phases.
Optimizers
Optimizer learning rates after WSD warm-up:
- Adam learning rate: 2e-4
- Muon learning rate: 8e-4
Muon was used to support larger critical batch sizes in a highly sparse MoE regime.
Infrastructure
- Hardware: 2,048 NVIDIA B300 GPUs
- Parallelism: HSDP + Expert Parallelism
- Compute partner: Prime Intellect
- Data partner: Datology
Intended Use
- Studying emergent behavior from large-scale pretraining
- Sparse MoE routing and load-balancing research
- Interpretability, probing, and ablation studies
- Domain-specific fine-tuning from a clean base
- Academic and industrial foundation model research
Rationale for Release
Most base model releases include instruction data, annealed training dynamics, or early alignment stages. Trinity-Large-TrueBase excludes these, providing an opportunity to study what large-scale models learn from pretraining data alone. This checkpoint is intended as a foundation for research rather than as a finished conversational assistant.
Known Limitations
- Not aligned for safety, helpfulness, or conversational tone
- Requires substantial compute and expertise to fine-tune
- May exhibit raw or unstable behaviors typical of unaligned models
- No extended-context tuning beyond the 8K pretraining window
License
Trinity-Large-TrueBase is released under the Apache License, Version 2.0.
Runs of arcee-ai Trinity-Large-TrueBase on huggingface.co
503
Total runs
0
24-hour runs
3
3-day runs
57
7-day runs
382
30-day runs
More Information About Trinity-Large-TrueBase huggingface.co Model
Url of Trinity-Large-TrueBase
Trinity-Large-TrueBase huggingface.co Url
Provider of Trinity-Large-TrueBase huggingface.co
Other API from arcee-ai
Updated:April 09 2026
Total runs:
19.9K
Run Growth:
3.9K
Growth Rate:
19.45%
Total runs:
18.7K
Run Growth:
2.4K
Growth Rate:
12.95%
Total runs:
10.0K
Run Growth:
9.3K
Growth Rate:
92.47%
Total runs:
7.4K
Run Growth:
-348
Growth Rate:
-4.72%
Total runs:
3.1K
Run Growth:
3.0K
Growth Rate:
99.61%
Total runs:
3.0K
Run Growth:
-33
Growth Rate:
-1.12%
Total runs:
2.2K
Run Growth:
641
Growth Rate:
29.26%
Total runs:
2.1K
Run Growth:
1.1K
Growth Rate:
53.31%
Total runs:
2.1K
Run Growth:
-299
Growth Rate:
-14.14%
Total runs:
1.8K
Run Growth:
1.5K
Growth Rate:
82.87%
Total runs:
1.5K
Run Growth:
1.5K
Growth Rate:
98.63%
Total runs:
1.4K
Run Growth:
17
Growth Rate:
1.20%
Total runs:
988
Run Growth:
-54
Growth Rate:
-5.46%
Total runs:
850
Run Growth:
0
Growth Rate:
0.00%
Total runs:
840
Run Growth:
631
Growth Rate:
75.12%
Total runs:
835
Run Growth:
342
Growth Rate:
46.15%
Total runs:
803
Run Growth:
-178
Growth Rate:
-21.84%
Total runs:
785
Run Growth:
-1.3K
Growth Rate:
-164.59%
Total runs:
779
Run Growth:
458
Growth Rate:
68.36%
Total runs:
766
Run Growth:
-14.7K
Growth Rate:
-1212.59%
Total runs:
758
Run Growth:
459
Growth Rate:
62.36%
Total runs:
614
Run Growth:
-4.4K
Growth Rate:
-715.15%
Total runs:
568
Run Growth:
318
Growth Rate:
57.30%
Total runs:
441
Run Growth:
159
Growth Rate:
36.05%
Total runs:
408
Run Growth:
0
Growth Rate:
0.00%
Total runs:
351
Run Growth:
351
Growth Rate:
100.00%
Total runs:
281
Run Growth:
0
Growth Rate:
0.00%
Total runs:
251
Run Growth:
159
Growth Rate:
65.16%
Total runs:
221
Run Growth:
153
Growth Rate:
69.23%
Total runs:
194
Run Growth:
-475
Growth Rate:
-238.69%
Total runs:
158
Run Growth:
3
Growth Rate:
1.88%
Total runs:
148
Run Growth:
-99
Growth Rate:
-66.89%
Total runs:
141
Run Growth:
43
Growth Rate:
30.50%
Total runs:
137
Run Growth:
115
Growth Rate:
83.94%
Total runs:
131
Run Growth:
16
Growth Rate:
12.12%
Total runs:
114
Run Growth:
-236
Growth Rate:
-207.02%
Total runs:
107
Run Growth:
1
Growth Rate:
0.93%
Total runs:
100
Run Growth:
79
Growth Rate:
79.00%
Total runs:
88
Run Growth:
-19
Growth Rate:
-20.88%
Total runs:
78
Run Growth:
-23
Growth Rate:
-29.49%
Total runs:
78
Run Growth:
-66
Growth Rate:
-84.62%
Total runs:
76
Run Growth:
74
Growth Rate:
94.87%
Total runs:
73
Run Growth:
62
Growth Rate:
84.93%
Total runs:
70
Run Growth:
-1
Growth Rate:
-1.43%
Total runs:
69
Run Growth:
-62
Growth Rate:
-131.91%
Total runs:
69
Run Growth:
51
Growth Rate:
73.91%
Total runs:
66
Run Growth:
-132
Growth Rate:
-200.00%
Total runs:
55
Run Growth:
-170
Growth Rate:
-309.09%
Total runs:
46
Run Growth:
32
Growth Rate:
69.57%
Total runs:
45
Run Growth:
35
Growth Rate:
77.78%
Total runs:
37
Run Growth:
8
Growth Rate:
21.62%
Total runs:
34
Run Growth:
29
Growth Rate:
82.86%
Total runs:
31
Run Growth:
-11
Growth Rate:
-35.48%
Total runs:
23
Run Growth:
9
Growth Rate:
39.13%
Total runs:
22
Run Growth:
13
Growth Rate:
54.17%
Total runs:
20
Run Growth:
0
Growth Rate:
0.00%
Total runs:
18
Run Growth:
-16
Growth Rate:
-80.00%
Total runs:
16
Run Growth:
11
Growth Rate:
64.71%
Total runs:
16
Run Growth:
-42
Growth Rate:
-262.50%
Total runs:
15
Run Growth:
-5
Growth Rate:
-33.33%
Total runs:
15
Run Growth:
-29
Growth Rate:
-181.25%
Total runs:
14
Run Growth:
-14
Growth Rate:
-127.27%
Total runs:
13
Run Growth:
6
Growth Rate:
50.00%
Total runs:
13
Run Growth:
9
Growth Rate:
64.29%
Total runs:
13
Run Growth:
4
Growth Rate:
28.57%
Total runs:
13
Run Growth:
9
Growth Rate:
69.23%
Total runs:
13
Run Growth:
12
Growth Rate:
92.31%
Total runs:
12
Run Growth:
6
Growth Rate:
50.00%
Total runs:
12
Run Growth:
9
Growth Rate:
64.29%
Total runs:
12
Run Growth:
-5
Growth Rate:
-41.67%