Total runs:
3.8K
Run Growth:
3.7K
Growth Rate:
97.26%
Sponsored by
APIMart
ToastyPigeon / tq14-gutenberg-test-ws
Model's Last Updated: January 30 2025
Introduction of tq14-gutenberg-test-ws
Model Details of tq14-gutenberg-test-ws
See axolotl config
axolotl version:
0.6.0
# git clone https://github.com/axolotl-ai-cloud/axolotl
# cd axolotl
# git checkout 3915abee4cba364a83f18cc4b8c2bb571cce3bac
# pip3 install packaging ninja huggingface_hub[cli]
# pip3 install -e '.[flash-attn,deepspeed]'
# apt update && apt install libopenmpi-dev
# pip install mpi4py
# huggingface-cli login --token $hf_key && wandb login $wandb_key
# python -m axolotl.cli.preprocess new-qwen-14-test.yml
# accelerate launch -m axolotl.cli.train new-qwen-14-test.yml
# python -m axolotl.cli.merge_lora new-qwen-14-test.yml
# huggingface-cli upload ToastyPigeon/tq14-unalign-test train-workspace/merged . --exclude "*.md"
# git clone https://github.com/axolotl-ai-cloud/axolotl && cd axolotl && pip3 install packaging ninja huggingface_hub[cli] && pip3 install -e '.[flash-attn,deepspeed]' && cd .. && huggingface-cli login --token $hf_key && wandb login $wandb_key
# Model
base_model: ToastyPigeon/Qwen2.5-14B-Instruct-1M-Unalign
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
load_in_4bit: true
strict: false
bf16: true
fp16:
tf32: false
flash_attention: true
special_tokens:
# Output
output_dir: ./train-workspace
hub_model_id: ToastyPigeon/tq14-gutenberg-test-ws
hub_strategy: "all_checkpoints"
resume_from_checkpoint:
saves_per_epoch: 4
# Data
sequence_len: 4096 # fits
min_sample_len: 128
dataset_prepared_path: last_run_prepared
datasets:
- path: ToastyPigeon/gutenberg-sft
type: chat_template
chat_template: chatml
field_messages: conversations
message_field_role: from
message_field_content: value
- path: sam-paech/gutenberg3-generalfiction-scifi-fantasy-romance-adventure-dpo
type: chat_template
chat_template: chatml
field_messages: chosen
message_field_role: role
message_field_content: content
warmup_steps: 20
shuffle_merged_datasets: true
sample_packing: true
pad_to_sequence_len: true
# Batching
num_epochs: 1
gradient_accumulation_steps: 1
micro_batch_size: 8
eval_batch_size: 1
# Evaluation
val_set_size: 50
evals_per_epoch: 10
eval_table_size:
eval_max_new_tokens: 256
eval_sample_packing: false
save_safetensors: true
# WandB
wandb_project: New-Qwen-Test
#wandb_entity:
gradient_checkpointing: 'unsloth'
gradient_checkpointing_kwargs:
use_reentrant: false
unsloth_cross_entropy_loss: true
#unsloth_lora_mlp: true
#unsloth_lora_qkv: true
#unsloth_lora_o: true
# LoRA
adapter: qlora
lora_model_dir:
lora_r: 32
lora_alpha: 64
lora_dropout: 0.125
lora_target_linear:
lora_fan_in_fan_out:
lora_target_modules:
- gate_proj
- down_proj
- up_proj
- q_proj
- v_proj
- k_proj
- o_proj
lora_modules_to_save:
#peft_use_dora: true
# Optimizer
optimizer: paged_ademamix_8bit # adamw_8bit
lr_scheduler: cosine
learning_rate: 5e-5
cosine_min_lr_ratio: 0.1
weight_decay: 0.01
max_grad_norm: 1.0
# Misc
train_on_inputs: false
group_by_length: false
early_stopping_patience:
local_rank:
logging_steps: 1
xformers_attention:
debug:
#deepspeed: /workspace/axolotl/deepspeed_configs/zero3_bf16.json # previously blank
fsdp:
fsdp_config:
plugins:
- axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_layer_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: true
gc_steps: 10
seed: 69
tq14-gutenberg-test-ws
This model is a fine-tuned version of ToastyPigeon/Qwen2.5-14B-Instruct-1M-Unalign on the ToastyPigeon/gutenberg-sft and the sam-paech/gutenberg3-generalfiction-scifi-fantasy-romance-adventure-dpo datasets. It achieves the following results on the evaluation set:
- Loss: 2.2556
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 1
- seed: 69
- optimizer: Use OptimizerNames.PAGED_ADEMAMIX_8BIT and the args are: No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 20
- num_epochs: 1
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 2.5227 | 0.0026 | 1 | 2.3420 |
| 2.389 | 0.1018 | 39 | 2.2861 |
| 2.358 | 0.2037 | 78 | 2.2738 |
| 2.4781 | 0.3055 | 117 | 2.2676 |
| 2.228 | 0.4073 | 156 | 2.2629 |
| 2.301 | 0.5091 | 195 | 2.2603 |
| 2.2135 | 0.6110 | 234 | 2.2583 |
| 2.3153 | 0.7128 | 273 | 2.2572 |
| 2.5095 | 0.8146 | 312 | 2.2561 |
| 2.3648 | 0.9164 | 351 | 2.2556 |
Framework versions
- PEFT 0.14.0
- Transformers 4.47.1
- Pytorch 2.5.1+cu124
- Datasets 3.1.0
- Tokenizers 0.21.0
Runs of ToastyPigeon tq14-gutenberg-test-ws on huggingface.co
16
Total runs
0
24-hour runs
10
3-day runs
9
7-day runs
10
30-day runs
More Information About tq14-gutenberg-test-ws huggingface.co Model
Url of tq14-gutenberg-test-ws
tq14-gutenberg-test-ws huggingface.co Url
Provider of tq14-gutenberg-test-ws huggingface.co
Other API from ToastyPigeon
Updated:May 19 2026
Total runs:
1.2K
Run Growth:
500
Growth Rate:
41.70%
Total runs:
1.2K
Run Growth:
961
Growth Rate:
80.42%
Total runs:
1.1K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
300
Run Growth:
0
Growth Rate:
0.00%
Total runs:
273
Run Growth:
0
Growth Rate:
0.00%
Total runs:
127
Run Growth:
0
Growth Rate:
0.00%
Total runs:
119
Run Growth:
40
Growth Rate:
33.61%
Total runs:
118
Run Growth:
118
Growth Rate:
100.00%
Total runs:
93
Run Growth:
5
Growth Rate:
5.38%
Total runs:
78
Run Growth:
10
Growth Rate:
12.82%
Total runs:
76
Run Growth:
3
Growth Rate:
3.95%
Total runs:
76
Run Growth:
76
Growth Rate:
100.00%
Total runs:
73
Run Growth:
-309
Growth Rate:
-423.29%
Total runs:
70
Run Growth:
67
Growth Rate:
93.06%
Total runs:
69
Run Growth:
69
Growth Rate:
100.00%
Total runs:
52
Run Growth:
0
Growth Rate:
0.00%
Total runs:
51
Run Growth:
17
Growth Rate:
33.33%
Total runs:
50
Run Growth:
-123
Growth Rate:
-246.00%
Total runs:
50
Run Growth:
-1
Growth Rate:
-2.00%
Total runs:
48
Run Growth:
45
Growth Rate:
93.75%
Total runs:
42
Run Growth:
0
Growth Rate:
0.00%
Total runs:
40
Run Growth:
35
Growth Rate:
87.50%
Total runs:
40
Run Growth:
0
Growth Rate:
0.00%
Total runs:
39
Run Growth:
-1.3K
Growth Rate:
-3238.46%
Total runs:
37
Run Growth:
0
Growth Rate:
0.00%
Total runs:
36
Run Growth:
0
Growth Rate:
0.00%
Total runs:
36
Run Growth:
0
Growth Rate:
0.00%
Total runs:
34
Run Growth:
0
Growth Rate:
0.00%
Total runs:
33
Run Growth:
-14
Growth Rate:
-42.42%
Total runs:
32
Run Growth:
32
Growth Rate:
100.00%
Total runs:
32
Run Growth:
21
Growth Rate:
70.00%
Total runs:
31
Run Growth:
18
Growth Rate:
58.06%
Total runs:
30
Run Growth:
30
Growth Rate:
100.00%
Total runs:
30
Run Growth:
27
Growth Rate:
90.00%
Total runs:
29
Run Growth:
4
Growth Rate:
13.79%
Total runs:
27
Run Growth:
0
Growth Rate:
0.00%
Total runs:
26
Run Growth:
0
Growth Rate:
0.00%
Total runs:
24
Run Growth:
9
Growth Rate:
37.50%
Total runs:
24
Run Growth:
0
Growth Rate:
0.00%
Total runs:
22
Run Growth:
-28
Growth Rate:
-127.27%
Total runs:
21
Run Growth:
-3
Growth Rate:
-14.29%
Total runs:
21
Run Growth:
1
Growth Rate:
4.76%
Total runs:
21
Run Growth:
0
Growth Rate:
0.00%
Total runs:
21
Run Growth:
21
Growth Rate:
100.00%
Total runs:
20
Run Growth:
0
Growth Rate:
0.00%
Total runs:
20
Run Growth:
20
Growth Rate:
100.00%
Total runs:
19
Run Growth:
0
Growth Rate:
0.00%
Total runs:
19
Run Growth:
0
Growth Rate:
0.00%
Total runs:
18
Run Growth:
6
Growth Rate:
33.33%
Total runs:
18
Run Growth:
-7
Growth Rate:
-41.18%
Total runs:
17
Run Growth:
6
Growth Rate:
35.29%
Total runs:
16
Run Growth:
-3
Growth Rate:
-18.75%
Total runs:
16
Run Growth:
3
Growth Rate:
18.75%
Total runs:
15
Run Growth:
2
Growth Rate:
13.33%
Total runs:
15
Run Growth:
5
Growth Rate:
33.33%
Total runs:
14
Run Growth:
11
Growth Rate:
78.57%
Total runs:
14
Run Growth:
8
Growth Rate:
57.14%
Total runs:
13
Run Growth:
-80
Growth Rate:
-615.38%
Total runs:
13
Run Growth:
0
Growth Rate:
0.00%
Total runs:
13
Run Growth:
4
Growth Rate:
28.57%
Total runs:
13
Run Growth:
0
Growth Rate:
0.00%
Total runs:
13
Run Growth:
13
Growth Rate:
100.00%
Total runs:
13
Run Growth:
9
Growth Rate:
69.23%
Total runs:
12
Run Growth:
5
Growth Rate:
41.67%
Total runs:
12
Run Growth:
0
Growth Rate:
0.00%
Total runs:
12
Run Growth:
6
Growth Rate:
50.00%
Total runs:
12
Run Growth:
8
Growth Rate:
66.67%
Total runs:
12
Run Growth:
9
Growth Rate:
75.00%
Total runs:
12
Run Growth:
-391
Growth Rate:
-3258.33%
Total runs:
12
Run Growth:
10
Growth Rate:
83.33%
Total runs:
11
Run Growth:
6
Growth Rate:
54.55%
Total runs:
11
Run Growth:
3
Growth Rate:
27.27%
Total runs:
11
Run Growth:
7
Growth Rate:
63.64%
Total runs:
11
Run Growth:
-18
Growth Rate:
-163.64%
Total runs:
11
Run Growth:
0
Growth Rate:
0.00%
Total runs:
11
Run Growth:
5
Growth Rate:
45.45%
Total runs:
10
Run Growth:
4
Growth Rate:
40.00%
Total runs:
10
Run Growth:
-2
Growth Rate:
-18.18%
Total runs:
10
Run Growth:
-6
Growth Rate:
-60.00%
Total runs:
10
Run Growth:
-4
Growth Rate:
-40.00%
Total runs:
10
Run Growth:
-1
Growth Rate:
-10.00%
Total runs:
10
Run Growth:
7
Growth Rate:
58.33%
Total runs:
10
Run Growth:
3
Growth Rate:
30.00%
Total runs:
10
Run Growth:
0
Growth Rate:
0.00%
Total runs:
9
Run Growth:
0
Growth Rate:
0.00%
Total runs:
9
Run Growth:
5
Growth Rate:
55.56%
Total runs:
9
Run Growth:
-1
Growth Rate:
-11.11%
Total runs:
9
Run Growth:
7
Growth Rate:
77.78%
Total runs:
9
Run Growth:
7
Growth Rate:
77.78%
Total runs:
9
Run Growth:
0
Growth Rate:
0.00%
Total runs:
9
Run Growth:
6
Growth Rate:
66.67%
Total runs:
9
Run Growth:
7
Growth Rate:
77.78%
Total runs:
9
Run Growth:
0
Growth Rate:
0.00%