Total runs:
2.2M
Run Growth:
-188
Growth Rate:
-0.01%
Sponsored by
APIMart
llm-jp / Dense-1.5B
Model's Last Updated: February 27 2025
Introduction of Dense-1.5B
Model Details of Dense-1.5B
Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
📄 [Paper] | 🤗 [Hugging Face] 📁 [Dataset] 💻 [Code] | 📊 [Log]
Model Index
We provide model checkpoints for all experiments to ensure reproducibility of the results presented in Tables 1 and 2.
Table 1
| Model | Link |
|---|---|
| 1 Dense 152M | Link |
| 2 MoE FS 8x152M | Link |
| 3 MoE BTX 8x152M | Link |
| 4 MoE NU 8x152M | Link |
| 5 MoE RNU (r=0.5) 8x152M | Link |
| 6 MoE DU (r=0.5) 8x152M | Link |
| 7 MoE DU (r=1.0) 8x152M | Link |
| 8 Dense 1.5B | Link |
| 9 MoE FS 8x1.5B | Link |
| 10 MoE BTX 8x1.5B | Link |
| 11 MoE NU 8x1.5B | Link |
| 12 MoE RNU (r=0.5) 8x1.5B | Link |
| 13 MoE DU (r=0.5) 8x1.5B | Link |
| 14 MoE DU (r=1.0) 8x1.5B | Link |
Table 2
| Model | Link |
|---|---|
| 1 Dense 3.7B | Link |
| 2 MoE FS 8x3.7B | Link |
| 3 MoE DU (r=0.5) 8x3.7B | Link |
| 4 Dense 13B | Link |
| 5 Dense 3.7B | Link |
BTX Experts
| Model | Link |
|---|---|
| Japanese expert 152M | Link |
| English expert 152M | Link |
| Code expert 152M | Link |
| Japanese expert 1.5B | Link |
| English expert 1.5B | Link |
| Code expert 1.5B | Link |
How to cite
If you find our work helpful, please feel free to cite.
@inproceedings{
nakamura2025dropupcycling,
title={Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization},
author={Taishi Nakamura and Takuya Akiba and Kazuki Fujii and Yusuke Oda and Rio Yokota and Jun Suzuki},
booktitle={The Thirteenth International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=gx1wHnf5Vp}
}
Runs of llm-jp Dense-1.5B on huggingface.co
90
Total runs
0
24-hour runs
5
3-day runs
19
7-day runs
75
30-day runs
More Information About Dense-1.5B huggingface.co Model
Url of Dense-1.5B
Dense-1.5B huggingface.co Url
Provider of Dense-1.5B huggingface.co
Other API from llm-jp
Updated:September 27 2024
Total runs:
39.5K
Run Growth:
13.2K
Growth Rate:
33.57%
Total runs:
34.4K
Run Growth:
33.7K
Growth Rate:
97.84%
Total runs:
30.7K
Run Growth:
1.1K
Growth Rate:
3.66%
Total runs:
7.2K
Run Growth:
1.3K
Growth Rate:
18.03%
Total runs:
6.1K
Run Growth:
-4.1K
Growth Rate:
-70.56%
Total runs:
5.9K
Run Growth:
5.4K
Growth Rate:
90.64%
Total runs:
5.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4.9K
Run Growth:
2.3K
Growth Rate:
47.90%
Total runs:
4.4K
Run Growth:
-588
Growth Rate:
-13.47%
Total runs:
4.4K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3.9K
Run Growth:
1.1K
Growth Rate:
28.51%
Total runs:
2.0K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.7K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.5K
Run Growth:
-319
Growth Rate:
-21.57%
Total runs:
1.3K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.3K
Run Growth:
-148
Growth Rate:
-11.56%
Total runs:
1.2K
Run Growth:
141
Growth Rate:
11.78%
Total runs:
1.2K
Run Growth:
-33
Growth Rate:
-2.78%
Total runs:
1.1K
Run Growth:
24
Growth Rate:
2.11%
Total runs:
999
Run Growth:
0
Growth Rate:
0.00%
Total runs:
887
Run Growth:
-447
Growth Rate:
-50.39%
Total runs:
772
Run Growth:
-33
Growth Rate:
-4.27%
Total runs:
590
Run Growth:
0
Growth Rate:
0.00%
Total runs:
536
Run Growth:
0
Growth Rate:
0.00%
Total runs:
518
Run Growth:
119
Growth Rate:
22.97%
Total runs:
500
Run Growth:
66
Growth Rate:
13.20%
Total runs:
449
Run Growth:
41
Growth Rate:
9.07%
Total runs:
445
Run Growth:
-13
Growth Rate:
-2.92%
Total runs:
294
Run Growth:
271
Growth Rate:
92.81%
Total runs:
212
Run Growth:
90
Growth Rate:
43.06%
Total runs:
197
Run Growth:
70
Growth Rate:
36.46%
Total runs:
190
Run Growth:
190
Growth Rate:
100.00%
Total runs:
180
Run Growth:
56
Growth Rate:
31.82%
Total runs:
164
Run Growth:
133
Growth Rate:
81.10%
Total runs:
151
Run Growth:
134
Growth Rate:
91.16%
Total runs:
135
Run Growth:
107
Growth Rate:
81.06%
Total runs:
133
Run Growth:
0
Growth Rate:
0.00%
Total runs:
128
Run Growth:
78
Growth Rate:
66.10%
Total runs:
121
Run Growth:
0
Growth Rate:
0.00%
Total runs:
112
Run Growth:
84
Growth Rate:
77.78%
Total runs:
112
Run Growth:
79
Growth Rate:
73.15%
Total runs:
112
Run Growth:
81
Growth Rate:
74.31%
Total runs:
107
Run Growth:
94
Growth Rate:
89.52%
Total runs:
98
Run Growth:
0
Growth Rate:
0.00%
Total runs:
96
Run Growth:
67
Growth Rate:
70.53%
Total runs:
94
Run Growth:
-46
Growth Rate:
-48.94%
Total runs:
92
Run Growth:
78
Growth Rate:
85.71%
Total runs:
91
Run Growth:
66
Growth Rate:
72.53%
Total runs:
89
Run Growth:
72
Growth Rate:
80.90%
Total runs:
84
Run Growth:
75
Growth Rate:
89.29%
Total runs:
83
Run Growth:
66
Growth Rate:
79.52%
Total runs:
71
Run Growth:
0
Growth Rate:
0.00%
Total runs:
54
Run Growth:
0
Growth Rate:
0.00%
Total runs:
40
Run Growth:
22
Growth Rate:
55.00%
Total runs:
38
Run Growth:
0
Growth Rate:
0.00%
Total runs:
23
Run Growth:
12
Growth Rate:
52.17%
Total runs:
14
Run Growth:
4
Growth Rate:
28.57%
Total runs:
14
Run Growth:
0
Growth Rate:
0.00%
Total runs:
12
Run Growth:
10
Growth Rate:
90.91%
Total runs:
12
Run Growth:
0
Growth Rate:
0.00%
Total runs:
10
Run Growth:
-3
Growth Rate:
-30.00%
Total runs:
8
Run Growth:
-6
Growth Rate:
-75.00%
Total runs:
8
Run Growth:
-1
Growth Rate:
-12.50%
Total runs:
5
Run Growth:
4
Growth Rate:
80.00%
Total runs:
3
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%