Total runs:
899
Run Growth:
-922
Growth Rate:
-102.56%
Sponsored by
Vmake Labs Video Enhancer
winglian / qwen3-4b-math-kd-jsd-temp1-v4
Model's Last Updated: June 03 2025
text-generation
Introduction of qwen3-4b-math-kd-jsd-temp1-v4
Model Details of qwen3-4b-math-kd-jsd-temp1-v4
See axolotl config
axolotl version:
0.10.0.dev0
base_model: winglian/qwen3-4b-math
plugins:
- axolotl.integrations.kd.KDPlugin
- axolotl.integrations.liger.LigerPlugin
liger_rms_norm: true
liger_glu_activation: true
# torch_compile: true
strict: false
kd_trainer: true
kd_ce_alpha: 0.1
kd_alpha: 1.0
kd_temperature: 1.0
kd_beta: 0.5
kd_normalize_topk: false
dataloader_prefetch_factor: 1
dataloader_num_workers: 2
dataloader_pin_memory: true
gc_steps: -1 # gc at the end of each epoch
chat_template: qwen3
datasets:
- path: winglian/OpenThoughts-114k-math-correct-qwen3-14b-math-prepared-temp1
type: chat_template
split: train
split_thinking: true
eot_tokens:
- "<|im_end|>"
skip_prepare_dataset: true
dataset_prepared_path: last_run_prepared
val_set_size: 0.0
output_dir: ./outputs/out-kd-4b-offline-t1-v4
sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true
wandb_project: kd-4b-math
wandb_entity: axolotl-ai
wandb_watch:
wandb_name:
wandb_log_model:
gradient_accumulation_steps: 1
micro_batch_size: 4
num_epochs: 2
optimizer: adamw_torch_fused
adam_beta2: 0.999
lr_scheduler: rex
learning_rate: 3e-5
max_grad_norm: 0.2
save_safetensors: true
bf16: true
tf32: true
gradient_checkpointing: true
gradient_checkpointing_kwargs:
use_reentrant: false
logging_steps: 1
flash_attention: true
warmup_steps: 100
evals_per_epoch: 4
saves_per_epoch: 2
debug:
weight_decay: 0.0
special_tokens:
eos_token: <|im_end|>
deepspeed: deepspeed_configs/zero2_torch_compile.json
outputs/out-kd-4b-offline-t1-v4
This model is a fine-tuned version of winglian/qwen3-4b-math on the winglian/OpenThoughts-114k-math-correct-qwen3-14b-math-prepared-temp1 dataset.
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- total_train_batch_size: 32
- total_eval_batch_size: 32
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
- num_epochs: 2.0
Training results
Framework versions
- Transformers 4.51.3
- Pytorch 2.7.0+cu128
- Datasets 3.5.1
- Tokenizers 0.21.1
Runs of winglian qwen3-4b-math-kd-jsd-temp1-v4 on huggingface.co
9
Total runs
0
24-hour runs
5
3-day runs
5
7-day runs
4
30-day runs
More Information About qwen3-4b-math-kd-jsd-temp1-v4 huggingface.co Model
Url of qwen3-4b-math-kd-jsd-temp1-v4
qwen3-4b-math-kd-jsd-temp1-v4 huggingface.co Url
Provider of qwen3-4b-math-kd-jsd-temp1-v4 huggingface.co
Other API from winglian
Updated:September 19 2023
Total runs:
822
Run Growth:
-910
Growth Rate:
-110.71%
Total runs:
813
Run Growth:
-919
Growth Rate:
-113.04%
Total runs:
729
Run Growth:
0
Growth Rate:
0.00%
Total runs:
187
Run Growth:
124
Growth Rate:
66.31%
Total runs:
153
Run Growth:
-667
Growth Rate:
-435.95%
Total runs:
75
Run Growth:
-1.4K
Growth Rate:
-1905.33%
Total runs:
26
Run Growth:
0
Growth Rate:
0.00%
Total runs:
24
Run Growth:
15
Growth Rate:
62.50%
Total runs:
24
Run Growth:
15
Growth Rate:
62.50%
Total runs:
24
Run Growth:
18
Growth Rate:
75.00%
Total runs:
21
Run Growth:
0
Growth Rate:
0.00%
Total runs:
16
Run Growth:
10
Growth Rate:
62.50%
Total runs:
15
Run Growth:
11
Growth Rate:
73.33%
Total runs:
15
Run Growth:
1
Growth Rate:
6.67%
Total runs:
14
Run Growth:
-9
Growth Rate:
-64.29%
Total runs:
14
Run Growth:
-19
Growth Rate:
-135.71%
Total runs:
12
Run Growth:
8
Growth Rate:
66.67%
Total runs:
12
Run Growth:
2
Growth Rate:
16.67%
Total runs:
12
Run Growth:
6
Growth Rate:
50.00%
Total runs:
11
Run Growth:
6
Growth Rate:
54.55%
Total runs:
10
Run Growth:
6
Growth Rate:
60.00%
Total runs:
10
Run Growth:
6
Growth Rate:
60.00%
Total runs:
10
Run Growth:
-5
Growth Rate:
-50.00%
Total runs:
9
Run Growth:
5
Growth Rate:
55.56%
Total runs:
8
Run Growth:
3
Growth Rate:
37.50%
Total runs:
8
Run Growth:
4
Growth Rate:
50.00%
Total runs:
6
Run Growth:
2
Growth Rate:
33.33%
Total runs:
6
Run Growth:
3
Growth Rate:
50.00%
Total runs:
6
Run Growth:
2
Growth Rate:
33.33%
Total runs:
6
Run Growth:
3
Growth Rate:
50.00%
Total runs:
6
Run Growth:
3
Growth Rate:
50.00%
Total runs:
5
Run Growth:
2
Growth Rate:
40.00%
Total runs:
5
Run Growth:
5
Growth Rate:
100.00%
Total runs:
5
Run Growth:
2
Growth Rate:
40.00%
Total runs:
4
Run Growth:
1
Growth Rate:
25.00%
Total runs:
3
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1
Run Growth:
-1
Growth Rate:
-100.00%
Total runs:
1
Run Growth:
-4
Growth Rate:
-400.00%
Total runs:
1
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
-5
Growth Rate:
0.00%
Total runs:
0
Run Growth:
-6
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
-1
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%