Total runs:
2.2M
Run Growth:
-188
Growth Rate:
-0.01%
Sponsored by
Vmake Labs Video Enhancer
llm-jp / Dense-13B
Model's Last Updated: February 27 2025
Introduction of Dense-13B
Model Details of Dense-13B
Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
📄 [Paper] | 🤗 [Hugging Face] 📁 [Dataset] 💻 [Code] | 📊 [Log]
Model Index
We provide model checkpoints for all experiments to ensure reproducibility of the results presented in Tables 1 and 2.
Table 1
| Model | Link |
|---|---|
| 1 Dense 152M | Link |
| 2 MoE FS 8x152M | Link |
| 3 MoE BTX 8x152M | Link |
| 4 MoE NU 8x152M | Link |
| 5 MoE RNU (r=0.5) 8x152M | Link |
| 6 MoE DU (r=0.5) 8x152M | Link |
| 7 MoE DU (r=1.0) 8x152M | Link |
| 8 Dense 1.5B | Link |
| 9 MoE FS 8x1.5B | Link |
| 10 MoE BTX 8x1.5B | Link |
| 11 MoE NU 8x1.5B | Link |
| 12 MoE RNU (r=0.5) 8x1.5B | Link |
| 13 MoE DU (r=0.5) 8x1.5B | Link |
| 14 MoE DU (r=1.0) 8x1.5B | Link |
Table 2
| Model | Link |
|---|---|
| 1 Dense 3.7B | Link |
| 2 MoE FS 8x3.7B | Link |
| 3 MoE DU (r=0.5) 8x3.7B | Link |
| 4 Dense 13B | Link |
| 5 Dense 3.7B | Link |
BTX Experts
| Model | Link |
|---|---|
| Japanese expert 152M | Link |
| English expert 152M | Link |
| Code expert 152M | Link |
| Japanese expert 1.5B | Link |
| English expert 1.5B | Link |
| Code expert 1.5B | Link |
How to cite
If you find our work helpful, please feel free to cite.
@inproceedings{
nakamura2025dropupcycling,
title={Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization},
author={Taishi Nakamura and Takuya Akiba and Kazuki Fujii and Yusuke Oda and Rio Yokota and Jun Suzuki},
booktitle={The Thirteenth International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=gx1wHnf5Vp}
}
Runs of llm-jp Dense-13B on huggingface.co
119
Total runs
8
24-hour runs
3
3-day runs
-2
7-day runs
32
30-day runs
More Information About Dense-13B huggingface.co Model
Url of Dense-13B
Dense-13B huggingface.co Url
Provider of Dense-13B huggingface.co
Other API from llm-jp
Updated:September 27 2024
Total runs:
39.5K
Run Growth:
13.2K
Growth Rate:
33.57%
Total runs:
34.4K
Run Growth:
33.7K
Growth Rate:
97.84%
Total runs:
30.7K
Run Growth:
1.1K
Growth Rate:
3.66%
Total runs:
8.6K
Run Growth:
8.6K
Growth Rate:
99.94%
Total runs:
7.2K
Run Growth:
1.3K
Growth Rate:
18.03%
Total runs:
6.5K
Run Growth:
-2.7K
Growth Rate:
-41.76%
Total runs:
5.9K
Run Growth:
5.4K
Growth Rate:
90.64%
Total runs:
5.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4.9K
Run Growth:
2.3K
Growth Rate:
47.90%
Total runs:
4.4K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4.3K
Run Growth:
4.2K
Growth Rate:
99.69%
Total runs:
2.0K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.6K
Run Growth:
1
Growth Rate:
0.06%
Total runs:
1.5K
Run Growth:
-319
Growth Rate:
-21.57%
Total runs:
1.4K
Run Growth:
1.3K
Growth Rate:
99.56%
Total runs:
1.3K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.3K
Run Growth:
-148
Growth Rate:
-11.56%
Total runs:
1.3K
Run Growth:
-71
Growth Rate:
-5.44%
Total runs:
1.2K
Run Growth:
141
Growth Rate:
11.78%
Total runs:
1.2K
Run Growth:
-33
Growth Rate:
-2.78%
Total runs:
1.1K
Run Growth:
24
Growth Rate:
2.11%
Total runs:
999
Run Growth:
0
Growth Rate:
0.00%
Total runs:
887
Run Growth:
-447
Growth Rate:
-50.39%
Total runs:
772
Run Growth:
-33
Growth Rate:
-4.27%
Total runs:
590
Run Growth:
0
Growth Rate:
0.00%
Total runs:
536
Run Growth:
0
Growth Rate:
0.00%
Total runs:
518
Run Growth:
119
Growth Rate:
22.97%
Total runs:
500
Run Growth:
66
Growth Rate:
13.20%
Total runs:
445
Run Growth:
-13
Growth Rate:
-2.92%
Total runs:
419
Run Growth:
-30
Growth Rate:
-7.16%
Total runs:
266
Run Growth:
68
Growth Rate:
25.56%
Total runs:
265
Run Growth:
83
Growth Rate:
31.32%
Total runs:
258
Run Growth:
89
Growth Rate:
34.77%
Total runs:
233
Run Growth:
110
Growth Rate:
47.21%
Total runs:
232
Run Growth:
110
Growth Rate:
47.83%
Total runs:
190
Run Growth:
190
Growth Rate:
100.00%
Total runs:
173
Run Growth:
37
Growth Rate:
21.39%
Total runs:
164
Run Growth:
133
Growth Rate:
81.10%
Total runs:
160
Run Growth:
68
Growth Rate:
42.50%
Total runs:
143
Run Growth:
41
Growth Rate:
28.67%
Total runs:
133
Run Growth:
0
Growth Rate:
0.00%
Total runs:
132
Run Growth:
27
Growth Rate:
20.45%
Total runs:
127
Run Growth:
21
Growth Rate:
16.54%
Total runs:
123
Run Growth:
-167
Growth Rate:
-135.77%
Total runs:
121
Run Growth:
0
Growth Rate:
0.00%
Total runs:
119
Run Growth:
31
Growth Rate:
26.05%
Total runs:
117
Run Growth:
25
Growth Rate:
21.37%
Total runs:
117
Run Growth:
29
Growth Rate:
24.79%
Total runs:
114
Run Growth:
25
Growth Rate:
21.93%
Total runs:
112
Run Growth:
30
Growth Rate:
26.79%
Total runs:
111
Run Growth:
31
Growth Rate:
27.93%
Total runs:
98
Run Growth:
0
Growth Rate:
0.00%
Total runs:
94
Run Growth:
-46
Growth Rate:
-48.94%
Total runs:
71
Run Growth:
0
Growth Rate:
0.00%
Total runs:
54
Run Growth:
0
Growth Rate:
0.00%
Total runs:
40
Run Growth:
22
Growth Rate:
55.00%
Total runs:
38
Run Growth:
0
Growth Rate:
0.00%
Total runs:
23
Run Growth:
12
Growth Rate:
52.17%
Total runs:
19
Run Growth:
10
Growth Rate:
52.63%
Total runs:
19
Run Growth:
14
Growth Rate:
73.68%
Total runs:
17
Run Growth:
7
Growth Rate:
41.18%
Total runs:
14
Run Growth:
4
Growth Rate:
28.57%
Total runs:
14
Run Growth:
0
Growth Rate:
0.00%
Total runs:
12
Run Growth:
0
Growth Rate:
0.00%
Total runs:
8
Run Growth:
-1
Growth Rate:
-12.50%
Total runs:
5
Run Growth:
4
Growth Rate:
80.00%
Total runs:
3
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%