Total runs:
1.2K
Run Growth:
1.1K
Growth Rate:
93.57%
Sponsored by
Vmake Labs Video Enhancer
AiAF / Mistral-QLoRA-Pretraining-Test-v1.1
Model's Last Updated: February 13 2025
Introduction of Mistral-QLoRA-Pretraining-Test-v1.1
Model Details of Mistral-QLoRA-Pretraining-Test-v1.1
See axolotl config
axolotl version:
0.6.0
base_model: mistralai/Mistral-7B-v0.1
# optionally might have model_type or tokenizer_type
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer
# Automatically upload checkpoint and final model to HF
hub_model_id: AiAF/Mistral-QLoRA-Pretraining-Test-v1.1
load_in_8bit: false
load_in_4bit: true
strict: false
datasets:
- path: AiAF/pretraining.jsonl
type: completion
dataset_prepared_path: last_run_prepared
val_set_size: 0.1
output_dir: /workspace/axolotl/outputs/qlora-out/Mistral-QLoRA-Pretraining-Test-V1.1.1
adapter: qlora
lora_model_dir:
sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true
lora_r: 8
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
lora_target_modules:
- gate_proj
- down_proj
- up_proj
- q_proj
- v_proj
- k_proj
- o_proj
wandb_project: "LLM_QLoRA-Pretraining-Practice"
wandb_entity:
wandb_watch: "all"
wandb_name: "Mistral-QLoRA-Pretraining-Test-V1.1"
wandb_log_model: "false"
gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 10
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.000005
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint: /workspace/axolotl/outputs/qlora-out/Mistral-QLoRA-Pretraining-Test-V1.1/checkpoint-40
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3
warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
Mistral-QLoRA-Pretraining-Test-v1.1
This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the AiAF/pretraining.jsonl dataset. It achieves the following results on the evaluation set:
- Loss: 1.8738
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 10.0
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.9972 | 0.1176 | 1 | 1.8782 |
| 1.8162 | 0.2353 | 2 | 1.8782 |
| 1.8588 | 0.4706 | 4 | 1.8783 |
| 2.0207 | 0.7059 | 6 | 1.8782 |
| 1.9881 | 0.9412 | 8 | 1.8780 |
| 1.9846 | 1.2353 | 10 | 1.8779 |
| 1.8436 | 1.4706 | 12 | 1.8778 |
| 1.9974 | 1.7059 | 14 | 1.8775 |
| 2.0703 | 1.9412 | 16 | 1.8773 |
| 1.9806 | 2.2353 | 18 | 1.8770 |
| 1.8501 | 2.4706 | 20 | 1.8769 |
| 1.9708 | 2.7059 | 22 | 1.8766 |
| 2.0717 | 2.9412 | 24 | 1.8763 |
| 2.126 | 3.2353 | 26 | 1.8762 |
| 1.931 | 3.4706 | 28 | 1.8760 |
| 1.8087 | 3.7059 | 30 | 1.8758 |
| 1.8101 | 3.9412 | 32 | 1.8758 |
| 2.0657 | 4.2353 | 34 | 1.8758 |
| 1.965 | 4.4706 | 36 | 1.8757 |
| 1.9222 | 4.7059 | 38 | 1.8757 |
| 1.9094 | 4.9412 | 40 | 1.8757 |
| 1.9283 | 5.2353 | 42 | 1.8756 |
| 2.0211 | 5.4706 | 44 | 1.8754 |
| 1.909 | 5.7059 | 46 | 1.8751 |
| 1.8289 | 5.9412 | 48 | 1.8749 |
| 1.9443 | 6.2353 | 50 | 1.8748 |
| 2.0195 | 6.4706 | 52 | 1.8747 |
| 1.7326 | 6.7059 | 54 | 1.8744 |
| 1.8524 | 6.9412 | 56 | 1.8743 |
| 1.958 | 7.2353 | 58 | 1.8742 |
| 1.9866 | 7.4706 | 60 | 1.8741 |
| 2.0558 | 7.7059 | 62 | 1.8741 |
| 1.9277 | 7.9412 | 64 | 1.8740 |
| 2.0108 | 8.2353 | 66 | 1.8739 |
| 1.9575 | 8.4706 | 68 | 1.8740 |
| 1.9107 | 8.7059 | 70 | 1.8739 |
| 1.9935 | 8.9412 | 72 | 1.8738 |
| 2.0618 | 9.2353 | 74 | 1.8738 |
| 1.8251 | 9.4706 | 76 | 1.8739 |
| 1.9817 | 9.7059 | 78 | 1.8739 |
| 1.9202 | 9.9412 | 80 | 1.8738 |
Framework versions
- PEFT 0.14.0
- Transformers 4.48.3
- Pytorch 2.5.1+cu124
- Datasets 3.2.0
- Tokenizers 0.21.0
Runs of AiAF Mistral-QLoRA-Pretraining-Test-v1.1 on huggingface.co
9
Total runs
-2
24-hour runs
-3
3-day runs
-13
7-day runs
-16
30-day runs
More Information About Mistral-QLoRA-Pretraining-Test-v1.1 huggingface.co Model
Url of Mistral-QLoRA-Pretraining-Test-v1.1
Mistral-QLoRA-Pretraining-Test-v1.1 huggingface.co Url
Provider of Mistral-QLoRA-Pretraining-Test-v1.1 huggingface.co
Other API from AiAF
Updated:May 08 2026
Total runs:
654
Run Growth:
55
Growth Rate:
9.00%
Total runs:
334
Run Growth:
128
Growth Rate:
39.14%
Total runs:
323
Run Growth:
-26
Growth Rate:
-8.36%
Total runs:
249
Run Growth:
134
Growth Rate:
57.76%
Total runs:
231
Run Growth:
-57
Growth Rate:
-24.89%
Total runs:
212
Run Growth:
168
Growth Rate:
80.00%
Total runs:
154
Run Growth:
58
Growth Rate:
40.00%
Total runs:
137
Run Growth:
2
Growth Rate:
1.48%
Total runs:
134
Run Growth:
22
Growth Rate:
16.79%
Total runs:
109
Run Growth:
26
Growth Rate:
24.30%
Total runs:
102
Run Growth:
11
Growth Rate:
11.11%
Total runs:
101
Run Growth:
67
Growth Rate:
69.07%
Total runs:
81
Run Growth:
-23
Growth Rate:
-33.33%
Total runs:
81
Run Growth:
-44
Growth Rate:
-55.00%
Total runs:
77
Run Growth:
-40
Growth Rate:
-52.63%
Total runs:
76
Run Growth:
-41
Growth Rate:
-52.56%
Total runs:
74
Run Growth:
-23
Growth Rate:
-32.86%
Total runs:
73
Run Growth:
31
Growth Rate:
43.66%
Total runs:
47
Run Growth:
30
Growth Rate:
75.00%
Total runs:
46
Run Growth:
-24
Growth Rate:
-51.06%
Total runs:
46
Run Growth:
46
Growth Rate:
100.00%
Total runs:
43
Run Growth:
21
Growth Rate:
47.73%
Total runs:
42
Run Growth:
0
Growth Rate:
0.00%
Total runs:
40
Run Growth:
19
Growth Rate:
47.50%
Total runs:
40
Run Growth:
4
Growth Rate:
11.11%
Total runs:
39
Run Growth:
14
Growth Rate:
41.18%
Total runs:
38
Run Growth:
18
Growth Rate:
47.37%
Total runs:
37
Run Growth:
-39
Growth Rate:
-108.33%
Total runs:
36
Run Growth:
20
Growth Rate:
58.82%
Total runs:
35
Run Growth:
-5
Growth Rate:
-14.29%
Total runs:
33
Run Growth:
11
Growth Rate:
33.33%
Total runs:
33
Run Growth:
-7
Growth Rate:
-21.88%
Total runs:
32
Run Growth:
24
Growth Rate:
72.73%
Total runs:
32
Run Growth:
-2
Growth Rate:
-6.25%
Total runs:
31
Run Growth:
-6
Growth Rate:
-18.18%
Total runs:
29
Run Growth:
-5
Growth Rate:
-18.52%
Total runs:
28
Run Growth:
-49
Growth Rate:
-188.46%
Total runs:
28
Run Growth:
15
Growth Rate:
53.57%
Total runs:
27
Run Growth:
0
Growth Rate:
0.00%
Total runs:
27
Run Growth:
-6
Growth Rate:
-24.00%
Total runs:
26
Run Growth:
2
Growth Rate:
7.41%
Total runs:
26
Run Growth:
19
Growth Rate:
73.08%
Total runs:
25
Run Growth:
17
Growth Rate:
68.00%
Total runs:
24
Run Growth:
-28
Growth Rate:
-112.00%
Total runs:
24
Run Growth:
-44
Growth Rate:
-191.30%
Total runs:
23
Run Growth:
11
Growth Rate:
47.83%
Total runs:
23
Run Growth:
0
Growth Rate:
0.00%
Total runs:
22
Run Growth:
15
Growth Rate:
71.43%
Total runs:
21
Run Growth:
7
Growth Rate:
33.33%
Total runs:
20
Run Growth:
0
Growth Rate:
0.00%
Total runs:
20
Run Growth:
-25
Growth Rate:
-113.64%
Total runs:
20
Run Growth:
4
Growth Rate:
20.00%
Total runs:
19
Run Growth:
12
Growth Rate:
60.00%
Total runs:
19
Run Growth:
10
Growth Rate:
52.63%
Total runs:
19
Run Growth:
15
Growth Rate:
71.43%
Total runs:
19
Run Growth:
-7
Growth Rate:
-41.18%
Total runs:
19
Run Growth:
12
Growth Rate:
63.16%
Total runs:
19
Run Growth:
2
Growth Rate:
10.53%
Total runs:
18
Run Growth:
0
Growth Rate:
0.00%
Total runs:
18
Run Growth:
7
Growth Rate:
35.00%
Total runs:
18
Run Growth:
9
Growth Rate:
47.37%
Total runs:
17
Run Growth:
-110
Growth Rate:
-687.50%
Total runs:
17
Run Growth:
11
Growth Rate:
64.71%
Total runs:
17
Run Growth:
12
Growth Rate:
66.67%
Total runs:
17
Run Growth:
-33
Growth Rate:
-183.33%
Total runs:
17
Run Growth:
-4
Growth Rate:
-23.53%
Total runs:
17
Run Growth:
-3
Growth Rate:
-17.65%
Total runs:
17
Run Growth:
4
Growth Rate:
26.67%
Total runs:
16
Run Growth:
-11
Growth Rate:
-57.89%
Total runs:
16
Run Growth:
-6
Growth Rate:
-40.00%
Total runs:
16
Run Growth:
13
Growth Rate:
81.25%
Total runs:
15
Run Growth:
2
Growth Rate:
12.50%
Total runs:
15
Run Growth:
-3
Growth Rate:
-20.00%
Total runs:
14
Run Growth:
14
Growth Rate:
93.33%
Total runs:
14
Run Growth:
10
Growth Rate:
71.43%
Total runs:
13
Run Growth:
-6
Growth Rate:
-46.15%
Total runs:
13
Run Growth:
-14
Growth Rate:
-100.00%
Total runs:
13
Run Growth:
6
Growth Rate:
46.15%
Total runs:
13
Run Growth:
5
Growth Rate:
27.78%
Total runs:
13
Run Growth:
5
Growth Rate:
38.46%
Total runs:
11
Run Growth:
-4
Growth Rate:
-44.44%
Total runs:
11
Run Growth:
-2
Growth Rate:
-18.18%
Total runs:
11
Run Growth:
1
Growth Rate:
9.09%
Total runs:
10
Run Growth:
-6
Growth Rate:
-60.00%
Total runs:
9
Run Growth:
-17
Growth Rate:
-188.89%