Sponsored by
APIMaster
mrfakename / train_test_step1
Model's Last Updated: December 18 2024
text-generation
Introduction of train_test_step1
Model Details of train_test_step1
See axolotl config
axolotl version:
0.6.0
base_model: /home/mainuser/workspace/cotmerge/out_sm
load_in_8bit: false
load_in_4bit: false
strict: false
plugins:
- axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: true
chat_template: chatml
datasets:
- path: mrfakename/datablend
split: train[:25%]
type: chat_template
field_messages: conversations
message_field_role: from
message_field_content: value
dataset_prepared_path: last_run_prepared
val_set_size: 0.05
output_dir: ./out_model
sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true
adapter:
lora_model_dir:
lora_r:
lora_alpha:
lora_dropout:
lora_target_linear:
lora_fan_in_fan_out:
wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
gradient_accumulation_steps: 8
micro_batch_size: 2
num_epochs: 1
optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 0.0002
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
flash_attn_cross_entropy: false
flash_attn_rms_norm: true
flash_attn_fuse_qkv: false
flash_attn_fuse_mlp: true
warmup_steps: 100
evals_per_epoch: 0
eval_table_size:
saves_per_epoch: 64
debug:
deepspeed:
deepspeed: /home/mainuser/axolotl/deepspeed_configs/zero2.json
weight_decay: 0.1
#fsdp:
# - full_shard
special_tokens:
bos_token: "<s>"
eos_token: "</s>"
unk_token: "<unk>"
tokens: # these are delimiters
- "<|im_start|>"
- "<|im_end|>"
out_model
This model was trained from scratch on the mrfakename/datablend dataset. It achieves the following results on the evaluation set:
- Loss: 0.9516
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- gradient_accumulation_steps: 8
- total_train_batch_size: 32
- total_eval_batch_size: 4
- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
- num_epochs: 1
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 7.2093 | 1.0 | 430 | 0.9516 |
Framework versions
- Transformers 4.47.0
- Pytorch 2.5.1+cu124
- Datasets 3.1.0
- Tokenizers 0.21.0
Runs of mrfakename train_test_step1 on huggingface.co
0
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs
More Information About train_test_step1 huggingface.co Model
Url of train_test_step1
train_test_step1 huggingface.co Url
Provider of train_test_step1 huggingface.co
Other API from mrfakename
Total runs:
353
Run Growth:
-1
Growth Rate:
-0.28%
Total runs:
327
Run Growth:
211
Growth Rate:
94.62%
Total runs:
288
Run Growth:
254
Growth Rate:
88.19%
Total runs:
209
Run Growth:
200
Growth Rate:
95.69%
Total runs:
151
Run Growth:
41
Growth Rate:
27.15%
Total runs:
115
Run Growth:
-64
Growth Rate:
-55.65%
Total runs:
99
Run Growth:
-303
Growth Rate:
-306.06%
Total runs:
87
Run Growth:
-48
Growth Rate:
-55.17%
Total runs:
86
Run Growth:
-217
Growth Rate:
-252.33%
Total runs:
74
Run Growth:
-16
Growth Rate:
-21.62%
Total runs:
62
Run Growth:
-43
Growth Rate:
-69.35%
Total runs:
62
Run Growth:
-15
Growth Rate:
-24.19%
Total runs:
50
Run Growth:
-28
Growth Rate:
-56.00%
Total runs:
45
Run Growth:
-81
Growth Rate:
-180.00%
Total runs:
45
Run Growth:
22
Growth Rate:
48.89%
Total runs:
26
Run Growth:
0
Growth Rate:
0.00%
Total runs:
24
Run Growth:
12
Growth Rate:
50.00%
Total runs:
21
Run Growth:
-36
Growth Rate:
-171.43%
Total runs:
21
Run Growth:
5
Growth Rate:
23.81%
Total runs:
15
Run Growth:
-13
Growth Rate:
-86.67%
Total runs:
14
Run Growth:
4
Growth Rate:
30.77%
Total runs:
13
Run Growth:
-2
Growth Rate:
-20.00%
Total runs:
13
Run Growth:
-5
Growth Rate:
-38.46%
Total runs:
13
Run Growth:
-4
Growth Rate:
-30.77%
Total runs:
12
Run Growth:
12
Growth Rate:
100.00%
Total runs:
11
Run Growth:
-1
Growth Rate:
-9.09%
Total runs:
11
Run Growth:
-1
Growth Rate:
-9.09%
Total runs:
11
Run Growth:
9
Growth Rate:
81.82%
Total runs:
10
Run Growth:
3
Growth Rate:
30.00%
Total runs:
9
Run Growth:
-3
Growth Rate:
-33.33%
Total runs:
9
Run Growth:
6
Growth Rate:
66.67%
Total runs:
9
Run Growth:
-1
Growth Rate:
-11.11%
Total runs:
9
Run Growth:
-8
Growth Rate:
-88.89%
Total runs:
7
Run Growth:
2
Growth Rate:
28.57%
Total runs:
6
Run Growth:
0
Growth Rate:
0.00%
Total runs:
6
Run Growth:
2
Growth Rate:
33.33%
Total runs:
6
Run Growth:
0
Growth Rate:
0.00%
Total runs:
6
Run Growth:
-5
Growth Rate:
-83.33%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4
Run Growth:
-5
Growth Rate:
-125.00%
Total runs:
4
Run Growth:
-4
Growth Rate:
-100.00%
Total runs:
4
Run Growth:
3
Growth Rate:
75.00%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3
Run Growth:
-2
Growth Rate:
-66.67%
Total runs:
2
Run Growth:
-1
Growth Rate:
-50.00%
Total runs:
2
Run Growth:
1
Growth Rate:
50.00%
Total runs:
2
Run Growth:
0
Growth Rate:
0.00%
Total runs:
2
Run Growth:
0
Growth Rate:
0.00%
Total runs:
2
Run Growth:
-1
Growth Rate:
-50.00%
Total runs:
2
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1
Run Growth:
-3
Growth Rate:
-300.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%