I messed up on this model
and forgot to add special tokens for <|im_start|> & <|im_end|> - but it still seems to work fine. If you're having any issues let me know in the Discussions tab and I can retrain it.
This model is usable, but you can get better results with the fixed latest model!
Check out the
latest model
instead of using this one!
A tiny 1B model that refuses basically anything you ask it! Trained on the
refusal
dataset. Prompt format is ChatML.
Training results:
|
Training Loss
|
Epoch
|
Step
|
Validation Loss
|
|
2.4352
|
0.0580
|
1
|
2.4462
|
|
1.5742
|
0.5217
|
9
|
1.4303
|
|
1.5084
|
1.0435
|
18
|
1.3672
|
|
1.0814
|
1.5217
|
27
|
1.3483
|
|
1.1024
|
2.0435
|
36
|
1.3204
|
|
0.6554
|
2.5217
|
45
|
1.4286
|
|
0.6163
|
3.0435
|
54
|
1.4375
|
|
0.5058
|
3.5072
|
63
|
1.4908
|
Training hyperparemeters:
The following hyperparameters were used during training:
-
learning_rate: 0.0002
-
train_batch_size: 2
-
eval_batch_size: 2
-
seed: 42
-
gradient_accumulation_steps: 4
-
total_train_batch_size: 8
-
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-
lr_scheduler_type: cosine
-
lr_scheduler_warmup_steps: 10
-
num_epochs: 4
Base model:
https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T