Total runs:
1.2K
Run Growth:
1.1K
Growth Rate:
93.95%
Sponsored by
APIMart
AiAF / Pretraining-SCPWiki-032025-12B-Instruct
Model's Last Updated: March 09 2025
text-generation
Introduction of Pretraining-SCPWiki-032025-12B-Instruct
Model Details of Pretraining-SCPWiki-032025-12B-Instruct
See axolotl config
axolotl version:
0.8.0.dev0
base_model: mistralai/Mistral-Nemo-Base-2407
# optionally might have model_type or tokenizer_type
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
# Automatically upload checkpoint and final model to HF
hub_model_id: AiAF/Pretraining-SCPWiki-032025-12B-Instruct
load_in_8bit: false
load_in_4bit: false
strict: false
datasets:
- path: AiAF/Pretraining-SCPWiki-032025-7B-Instruct-pretraining.jsonl
type: completion
dataset_prepared_path: last_run_prepared
val_set_size: 0.1
output_dir: ./outputs/out/Pretraining-SCPWiki-032025-12B-V1
sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true
eval_sample_packing: false
wandb_project: "LLM-Pretraining"
wandb_entity:
wandb_watch: "all"
wandb_name: "Pretraining-SCPWiki-032025-12B-V1"
wandb_log_model: "false"
gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 1
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.000005
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3
save_total_limit: 30
warmup_steps: 10
evals_per_epoch: 20
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 20
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
eos_token: "<|im_end|>"
pad_token: "<pad>"
bos_token: "<s>"
unk_token: "<unk>"
tokens:
- "<|im_start|>"
Pretraining-SCPWiki-032025-12B-Instruct
This model is a fine-tuned version of mistralai/Mistral-Nemo-Base-2407 on the AiAF/Pretraining-SCPWiki-032025-7B-Instruct-pretraining.jsonl dataset. It achieves the following results on the evaluation set:
- Loss: 1.5467
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 1.0
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 3.1576 | 0.0018 | 1 | 3.5143 |
| 1.4459 | 0.0511 | 29 | 1.6213 |
| 1.4502 | 0.1022 | 58 | 1.6003 |
| 1.5545 | 0.1534 | 87 | 1.5870 |
| 1.3624 | 0.2045 | 116 | 1.5779 |
| 1.3053 | 0.2556 | 145 | 1.5691 |
| 1.5688 | 0.3067 | 174 | 1.5635 |
| 1.7144 | 0.3579 | 203 | 1.5594 |
| 1.5199 | 0.4090 | 232 | 1.5550 |
| 1.2483 | 0.4601 | 261 | 1.5516 |
| 1.4053 | 0.5112 | 290 | 1.5493 |
| 1.4238 | 0.5624 | 319 | 1.5486 |
| 1.4939 | 0.6135 | 348 | 1.5477 |
| 1.4072 | 0.6646 | 377 | 1.5472 |
| 1.6039 | 0.7157 | 406 | 1.5469 |
| 1.3127 | 0.7669 | 435 | 1.5468 |
| 1.4754 | 0.8180 | 464 | 1.5466 |
| 1.5992 | 0.8691 | 493 | 1.5467 |
| 1.421 | 0.9202 | 522 | 1.5467 |
| 1.2666 | 0.9714 | 551 | 1.5467 |
Framework versions
- Transformers 4.49.0
- Pytorch 2.5.1+cu124
- Datasets 3.2.0
- Tokenizers 0.21.0
Runs of AiAF Pretraining-SCPWiki-032025-12B-Instruct on huggingface.co
19
Total runs
0
24-hour runs
7
3-day runs
4
7-day runs
9
30-day runs
More Information About Pretraining-SCPWiki-032025-12B-Instruct huggingface.co Model
Url of Pretraining-SCPWiki-032025-12B-Instruct
Pretraining-SCPWiki-032025-12B-Instruct huggingface.co Url
Provider of Pretraining-SCPWiki-032025-12B-Instruct huggingface.co
Other API from AiAF
Updated:May 08 2026
Total runs:
611
Run Growth:
70
Growth Rate:
11.40%
Total runs:
327
Run Growth:
130
Growth Rate:
40.63%
Total runs:
311
Run Growth:
-24
Growth Rate:
-7.67%
Total runs:
232
Run Growth:
145
Growth Rate:
62.50%
Total runs:
229
Run Growth:
-50
Growth Rate:
-21.65%
Total runs:
145
Run Growth:
57
Growth Rate:
42.54%
Total runs:
135
Run Growth:
-4
Growth Rate:
-3.08%
Total runs:
131
Run Growth:
31
Growth Rate:
23.48%
Total runs:
107
Run Growth:
24
Growth Rate:
22.43%
Total runs:
99
Run Growth:
7
Growth Rate:
7.37%
Total runs:
97
Run Growth:
65
Growth Rate:
72.22%
Total runs:
80
Run Growth:
-49
Growth Rate:
-62.03%
Total runs:
78
Run Growth:
-40
Growth Rate:
-51.28%
Total runs:
76
Run Growth:
-38
Growth Rate:
-50.00%
Total runs:
71
Run Growth:
35
Growth Rate:
47.95%
Total runs:
70
Run Growth:
-26
Growth Rate:
-39.39%
Total runs:
69
Run Growth:
-28
Growth Rate:
-45.90%
Total runs:
47
Run Growth:
-33
Growth Rate:
-70.21%
Total runs:
46
Run Growth:
46
Growth Rate:
100.00%
Total runs:
44
Run Growth:
21
Growth Rate:
47.73%
Total runs:
40
Run Growth:
13
Growth Rate:
36.11%
Total runs:
40
Run Growth:
27
Growth Rate:
72.97%
Total runs:
38
Run Growth:
15
Growth Rate:
41.67%
Total runs:
38
Run Growth:
2
Growth Rate:
5.41%
Total runs:
36
Run Growth:
-37
Growth Rate:
-90.24%
Total runs:
36
Run Growth:
10
Growth Rate:
25.64%
Total runs:
35
Run Growth:
-4
Growth Rate:
-11.76%
Total runs:
34
Run Growth:
13
Growth Rate:
40.63%
Total runs:
34
Run Growth:
15
Growth Rate:
46.88%
Total runs:
33
Run Growth:
9
Growth Rate:
23.68%
Total runs:
33
Run Growth:
9
Growth Rate:
27.27%
Total runs:
33
Run Growth:
25
Growth Rate:
75.76%
Total runs:
32
Run Growth:
-2
Growth Rate:
-6.06%
Total runs:
32
Run Growth:
-7
Growth Rate:
-20.59%
Total runs:
28
Run Growth:
17
Growth Rate:
60.71%
Total runs:
27
Run Growth:
5
Growth Rate:
18.52%
Total runs:
27
Run Growth:
-6
Growth Rate:
-22.22%
Total runs:
26
Run Growth:
-52
Growth Rate:
-236.36%
Total runs:
26
Run Growth:
19
Growth Rate:
73.08%
Total runs:
25
Run Growth:
-30
Growth Rate:
-120.00%
Total runs:
25
Run Growth:
-5
Growth Rate:
-20.00%
Total runs:
25
Run Growth:
18
Growth Rate:
72.00%
Total runs:
24
Run Growth:
-2
Growth Rate:
-9.09%
Total runs:
23
Run Growth:
-43
Growth Rate:
-195.45%
Total runs:
23
Run Growth:
11
Growth Rate:
47.83%
Total runs:
23
Run Growth:
0
Growth Rate:
0.00%
Total runs:
22
Run Growth:
-25
Growth Rate:
-119.05%
Total runs:
21
Run Growth:
5
Growth Rate:
45.45%
Total runs:
21
Run Growth:
6
Growth Rate:
30.00%
Total runs:
21
Run Growth:
18
Growth Rate:
81.82%
Total runs:
20
Run Growth:
0
Growth Rate:
0.00%
Total runs:
20
Run Growth:
3
Growth Rate:
27.27%
Total runs:
20
Run Growth:
11
Growth Rate:
55.00%
Total runs:
20
Run Growth:
8
Growth Rate:
36.36%
Total runs:
19
Run Growth:
-10
Growth Rate:
-50.00%
Total runs:
19
Run Growth:
9
Growth Rate:
50.00%
Total runs:
19
Run Growth:
12
Growth Rate:
63.16%
Total runs:
19
Run Growth:
3
Growth Rate:
15.00%
Total runs:
18
Run Growth:
12
Growth Rate:
66.67%
Total runs:
18
Run Growth:
-33
Growth Rate:
-183.33%
Total runs:
18
Run Growth:
6
Growth Rate:
33.33%
Total runs:
17
Run Growth:
1
Growth Rate:
14.29%
Total runs:
17
Run Growth:
2
Growth Rate:
10.53%
Total runs:
17
Run Growth:
-4
Growth Rate:
-23.53%
Total runs:
17
Run Growth:
-8
Growth Rate:
-50.00%
Total runs:
17
Run Growth:
1
Growth Rate:
5.88%
Total runs:
16
Run Growth:
-108
Growth Rate:
-675.00%
Total runs:
16
Run Growth:
4
Growth Rate:
25.00%
Total runs:
16
Run Growth:
13
Growth Rate:
81.25%
Total runs:
15
Run Growth:
5
Growth Rate:
83.33%
Total runs:
15
Run Growth:
-4
Growth Rate:
-26.67%
Total runs:
15
Run Growth:
1
Growth Rate:
8.33%
Total runs:
15
Run Growth:
-6
Growth Rate:
-40.00%
Total runs:
14
Run Growth:
-12
Growth Rate:
-85.71%
Total runs:
14
Run Growth:
0
Growth Rate:
0.00%
Total runs:
13
Run Growth:
-5
Growth Rate:
-38.46%
Total runs:
13
Run Growth:
4
Growth Rate:
36.36%
Total runs:
13
Run Growth:
5
Growth Rate:
38.46%
Total runs:
11
Run Growth:
-1
Growth Rate:
-9.09%
Total runs:
11
Run Growth:
3
Growth Rate:
23.08%
Total runs:
10
Run Growth:
-7
Growth Rate:
-70.00%
Total runs:
9
Run Growth:
-6
Growth Rate:
-66.67%
Total runs:
9
Run Growth:
-14
Growth Rate:
-127.27%