Total runs:
2.2M
Run Growth:
-188
Growth Rate:
-0.01%
Sponsored by
iMean AI Trip Planner
llm-jp / Dense-3.7B
Model's Last Updated: February 27 2025
Introduction of Dense-3.7B
Model Details of Dense-3.7B
Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
📄 [Paper] | 🤗 [Hugging Face] 📁 [Dataset] 💻 [Code] | 📊 [Log]
Model Index
We provide model checkpoints for all experiments to ensure reproducibility of the results presented in Tables 1 and 2.
Table 1
| Model | Link |
|---|---|
| 1 Dense 152M | Link |
| 2 MoE FS 8x152M | Link |
| 3 MoE BTX 8x152M | Link |
| 4 MoE NU 8x152M | Link |
| 5 MoE RNU (r=0.5) 8x152M | Link |
| 6 MoE DU (r=0.5) 8x152M | Link |
| 7 MoE DU (r=1.0) 8x152M | Link |
| 8 Dense 1.5B | Link |
| 9 MoE FS 8x1.5B | Link |
| 10 MoE BTX 8x1.5B | Link |
| 11 MoE NU 8x1.5B | Link |
| 12 MoE RNU (r=0.5) 8x1.5B | Link |
| 13 MoE DU (r=0.5) 8x1.5B | Link |
| 14 MoE DU (r=1.0) 8x1.5B | Link |
Table 2
| Model | Link |
|---|---|
| 1 Dense 3.7B | Link |
| 2 MoE FS 8x3.7B | Link |
| 3 MoE DU (r=0.5) 8x3.7B | Link |
| 4 Dense 13B | Link |
| 5 Dense 3.7B | Link |
BTX Experts
| Model | Link |
|---|---|
| Japanese expert 152M | Link |
| English expert 152M | Link |
| Code expert 152M | Link |
| Japanese expert 1.5B | Link |
| English expert 1.5B | Link |
| Code expert 1.5B | Link |
How to cite
If you find our work helpful, please feel free to cite.
@inproceedings{
nakamura2025dropupcycling,
title={Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization},
author={Taishi Nakamura and Takuya Akiba and Kazuki Fujii and Yusuke Oda and Rio Yokota and Jun Suzuki},
booktitle={The Thirteenth International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=gx1wHnf5Vp}
}
Runs of llm-jp Dense-3.7B on huggingface.co
102
Total runs
0
24-hour runs
3
3-day runs
13
7-day runs
57
30-day runs
More Information About Dense-3.7B huggingface.co Model
Url of Dense-3.7B
Dense-3.7B huggingface.co Url
Provider of Dense-3.7B huggingface.co
Other API from llm-jp
Updated:September 27 2024
Total runs:
39.5K
Run Growth:
13.2K
Growth Rate:
33.57%
Total runs:
34.4K
Run Growth:
33.7K
Growth Rate:
97.84%
Total runs:
30.7K
Run Growth:
1.1K
Growth Rate:
3.66%
Total runs:
7.2K
Run Growth:
1.3K
Growth Rate:
18.03%
Total runs:
6.5K
Run Growth:
-2.7K
Growth Rate:
-41.76%
Total runs:
5.9K
Run Growth:
5.4K
Growth Rate:
90.64%
Total runs:
5.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4.9K
Run Growth:
2.3K
Growth Rate:
47.90%
Total runs:
4.4K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3.9K
Run Growth:
1.1K
Growth Rate:
28.51%
Total runs:
2.7K
Run Growth:
2.4K
Growth Rate:
99.47%
Total runs:
2.3K
Run Growth:
2.2K
Growth Rate:
99.77%
Total runs:
2.0K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.6K
Run Growth:
1
Growth Rate:
0.06%
Total runs:
1.5K
Run Growth:
-319
Growth Rate:
-21.57%
Total runs:
1.4K
Run Growth:
1.2K
Growth Rate:
99.52%
Total runs:
1.3K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.3K
Run Growth:
-148
Growth Rate:
-11.56%
Total runs:
1.2K
Run Growth:
141
Growth Rate:
11.78%
Total runs:
1.2K
Run Growth:
-33
Growth Rate:
-2.78%
Total runs:
1.1K
Run Growth:
24
Growth Rate:
2.11%
Total runs:
999
Run Growth:
0
Growth Rate:
0.00%
Total runs:
887
Run Growth:
-447
Growth Rate:
-50.39%
Total runs:
772
Run Growth:
-33
Growth Rate:
-4.27%
Total runs:
590
Run Growth:
0
Growth Rate:
0.00%
Total runs:
536
Run Growth:
0
Growth Rate:
0.00%
Total runs:
518
Run Growth:
119
Growth Rate:
22.97%
Total runs:
500
Run Growth:
66
Growth Rate:
13.20%
Total runs:
445
Run Growth:
-13
Growth Rate:
-2.92%
Total runs:
435
Run Growth:
56
Growth Rate:
12.44%
Total runs:
265
Run Growth:
114
Growth Rate:
43.51%
Total runs:
262
Run Growth:
121
Growth Rate:
46.90%
Total runs:
257
Run Growth:
127
Growth Rate:
50.40%
Total runs:
218
Run Growth:
143
Growth Rate:
66.51%
Total runs:
214
Run Growth:
167
Growth Rate:
79.52%
Total runs:
209
Run Growth:
110
Growth Rate:
52.38%
Total runs:
190
Run Growth:
190
Growth Rate:
100.00%
Total runs:
164
Run Growth:
133
Growth Rate:
81.10%
Total runs:
153
Run Growth:
109
Growth Rate:
70.32%
Total runs:
149
Run Growth:
89
Growth Rate:
57.42%
Total runs:
143
Run Growth:
80
Growth Rate:
55.17%
Total runs:
138
Run Growth:
75
Growth Rate:
53.19%
Total runs:
133
Run Growth:
0
Growth Rate:
0.00%
Total runs:
132
Run Growth:
-114
Growth Rate:
-85.07%
Total runs:
121
Run Growth:
0
Growth Rate:
0.00%
Total runs:
120
Run Growth:
52
Growth Rate:
43.33%
Total runs:
117
Run Growth:
64
Growth Rate:
54.24%
Total runs:
113
Run Growth:
60
Growth Rate:
53.10%
Total runs:
112
Run Growth:
63
Growth Rate:
56.25%
Total runs:
111
Run Growth:
66
Growth Rate:
59.46%
Total runs:
107
Run Growth:
60
Growth Rate:
56.07%
Total runs:
98
Run Growth:
0
Growth Rate:
0.00%
Total runs:
94
Run Growth:
-46
Growth Rate:
-48.94%
Total runs:
71
Run Growth:
0
Growth Rate:
0.00%
Total runs:
54
Run Growth:
0
Growth Rate:
0.00%
Total runs:
40
Run Growth:
22
Growth Rate:
55.00%
Total runs:
38
Run Growth:
0
Growth Rate:
0.00%
Total runs:
23
Run Growth:
19
Growth Rate:
82.61%
Total runs:
23
Run Growth:
12
Growth Rate:
52.17%
Total runs:
15
Run Growth:
7
Growth Rate:
46.67%
Total runs:
14
Run Growth:
4
Growth Rate:
28.57%
Total runs:
14
Run Growth:
-1
Growth Rate:
-7.14%
Total runs:
14
Run Growth:
0
Growth Rate:
0.00%
Total runs:
12
Run Growth:
0
Growth Rate:
0.00%
Total runs:
8
Run Growth:
-1
Growth Rate:
-12.50%
Total runs:
5
Run Growth:
4
Growth Rate:
80.00%
Total runs:
3
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%