Total runs:
1.2K
Run Growth:
-972
Growth Rate:
-82.16%
Sponsored by
PostMcpAi
Mxode / NanoLM-365M-Base
Model's Last Updated: September 08 2024
text-generation
Introduction of NanoLM-365M-Base
Model Details of NanoLM-365M-Base
NanoLM-365M-base
English | 简体中文
Introduction
Based on Qwen2-0.5B , the tokenizer has been replaced with BilingualTokenizer-8K to reduce the number of parameters. The total parameters have been reduced from 0.5B to 365M.
Details
To recover some performance and facilitate fine-tuning for downstream tasks, I chose to freeze the backbone parameters and only train the embedding part after replacing the tokenizer. Training was conducted for 40,000 steps on wikipedia-zh and cosmopedia-100k .
| Value | |
|---|---|
| Total Params | 365 M |
| Trainable Params | < 10 M |
| Trainable Parts |
model.embed_tokens
|
| Training Steps | 40,000 |
| Training Dataset | wikipedia-zh , cosmopedia-100k |
| Optimizer | adamw_torch |
| Learning Rate | 2e-4 |
| LR Scheduler | cosine |
| Weight Decay | 0.1 |
| Warm-up Ratio | 0.03 |
| Batch Size | 16 |
| Gradient Accumulation Steps | 1 |
| Seq Len | 4096 |
| Dtype | bf16 |
| Peak GPU Memory | < 48 GB |
| Device | NVIDIA A100-SXM4-80GB |
The specific training records are as follows:
Runs of Mxode NanoLM-365M-Base on huggingface.co
7
Total runs
0
24-hour runs
0
3-day runs
-1
7-day runs
5
30-day runs
More Information About NanoLM-365M-Base huggingface.co Model
Url of NanoLM-365M-Base
NanoLM-365M-Base huggingface.co Url
Provider of NanoLM-365M-Base huggingface.co
Other API from Mxode
Updated:April 28 2024
Total runs:
73
Run Growth:
27
Growth Rate:
36.99%
Total runs:
51
Run Growth:
-789
Growth Rate:
-1547.06%
Total runs:
25
Run Growth:
20
Growth Rate:
80.00%
Total runs:
21
Run Growth:
-2
Growth Rate:
-9.52%
Total runs:
21
Run Growth:
-6
Growth Rate:
-28.57%
Total runs:
21
Run Growth:
13
Growth Rate:
61.90%
Total runs:
19
Run Growth:
-3
Growth Rate:
-15.79%
Total runs:
19
Run Growth:
11
Growth Rate:
57.89%
Total runs:
18
Run Growth:
14
Growth Rate:
77.78%
Total runs:
16
Run Growth:
8
Growth Rate:
50.00%
Total runs:
14
Run Growth:
9
Growth Rate:
64.29%
Total runs:
13
Run Growth:
-5
Growth Rate:
-38.46%
Total runs:
13
Run Growth:
2
Growth Rate:
15.38%
Total runs:
12
Run Growth:
8
Growth Rate:
66.67%
Total runs:
11
Run Growth:
-14
Growth Rate:
-127.27%
Total runs:
3
Run Growth:
1
Growth Rate:
33.33%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%