Total runs:
15.4K
Run Growth:
0
Growth Rate:
0.00%
Sponsored by
Agentkit
minpeter / QLoRA-Llama-3.2-1B-alpaca
Model's Last Updated: February 05 2025
Introduction of QLoRA-Llama-3.2-1B-alpaca
Model Details of QLoRA-Llama-3.2-1B-alpaca
See axolotl config
axolotl version:
0.6.0
base_model: meta-llama/Llama-3.2-1B
# Automatically upload checkpoint and final model to HF
# hub_model_id: username/custom_model_name
load_in_8bit: false
load_in_4bit: true
strict: false
datasets:
- path: yahma/alpaca-cleaned
type: alpaca
dataset_prepared_path: last_run_prepared
val_set_size: 0.1
output_dir: ./outputs/qlora-out
adapter: qlora
lora_model_dir:
sequence_len: 2048
sample_packing: true
eval_sample_packing: true
pad_to_sequence_len: true
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_fan_in_fan_out:
lora_target_modules:
- gate_proj
- down_proj
- up_proj
- q_proj
- v_proj
- k_proj
- o_proj
wandb_project: "axolotl"
wandb_entity: "kasfiekfs-e"
wandb_watch:
wandb_name:
wandb_log_model:
gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 1
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3
warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
pad_token: "<|end_of_text|>"
QLoRA-Llama-3.2-1B-Alpaca
This model is a fine-tuned version of meta-llama/Llama-3.2-1B on the yahma/alpaca-cleaned dataset. It achieves the following results on the evaluation set:
- Loss: 1.2517
Model description
This is an instruction-tuned model trained using the Alpaca Prompt Format with the cleaned version of the original Alpaca Dataset (yahma/alpaca-cleaned) released by Stanford.
Please note that this model is based on the Llama 3.2 1B Base (pretrained) model.
Intended uses & limitations
The following chat template provides optimal performance for inference:
# w/o train system prompt
{{ bos_token }}{% for message in messages %}{% if message['role'] == 'user' %}{{ '### Instruction:\n' + message['content'] + '\n\n' }}{% elif message['role'] == 'assistant' %}{{ '### Response:\n' + message['content'] + eos_token + '\n\n' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '### Response:\n' }}{% endif %}
# w/ train system prompt
{{ bos_token }}{{ 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n' }}{% for message in messages %}{% if message['role'] == 'user' %}{{ '### Instruction:\n' + message['content'] + '\n\n' }}{% elif message['role'] == 'assistant' %}{{ '### Response:\n' + message['content'] + eos_token + '\n\n' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '### Response:\n' }}{% endif %}
The model was trained with the system prompt:
"Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request."
However, it has not been tested whether including this prompt in inference is the optimal choice. Feel free to experiment.
An example rendering with the
w/ train system prompt
template is as follows:
<|begin_of_text|>### Instruction:
what is 43 + 12?
### Response:
The sum of numbers 43 and 12 is 55.<|end_of_text|>
In this case, the model output is:
The sum of numbers 43 and 12 is 55.<|end_of_text|>
Be mindful of the EOS token.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 1.0
Training results
https://wandb.ai/kasfiekfs-e/axolotl/runs/uzitnvvj
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.4939 | 0.0018 | 1 | 1.4682 |
| 1.3169 | 0.2510 | 138 | 1.2773 |
| 1.2603 | 0.5020 | 276 | 1.2603 |
| 1.2301 | 0.7531 | 414 | 1.2517 |
Framework versions
- PEFT 0.14.0
- Transformers 4.47.1
- Pytorch 2.5.1+cu124
- Datasets 3.2.0
- Tokenizers 0.21.0
Runs of minpeter QLoRA-Llama-3.2-1B-alpaca on huggingface.co
30
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs
More Information About QLoRA-Llama-3.2-1B-alpaca huggingface.co Model
Url of QLoRA-Llama-3.2-1B-alpaca
QLoRA-Llama-3.2-1B-alpaca huggingface.co Url
Provider of QLoRA-Llama-3.2-1B-alpaca huggingface.co
Other API from minpeter
Updated:July 30 2025
Total runs:
739
Run Growth:
0
Growth Rate:
0.00%
Total runs:
559
Run Growth:
-27
Growth Rate:
-4.83%
Total runs:
238
Run Growth:
0
Growth Rate:
0.00%
Total runs:
193
Run Growth:
148
Growth Rate:
71.50%
Total runs:
141
Run Growth:
0
Growth Rate:
0.00%
Total runs:
131
Run Growth:
0
Growth Rate:
0.00%
Total runs:
111
Run Growth:
4
Growth Rate:
3.60%
Total runs:
110
Run Growth:
3
Growth Rate:
2.73%
Total runs:
79
Run Growth:
21
Growth Rate:
26.58%
Total runs:
78
Run Growth:
9
Growth Rate:
11.54%
Total runs:
63
Run Growth:
-23
Growth Rate:
-36.51%
Total runs:
49
Run Growth:
0
Growth Rate:
0.00%
Total runs:
39
Run Growth:
16
Growth Rate:
41.03%
Total runs:
37
Run Growth:
10
Growth Rate:
27.03%
Total runs:
29
Run Growth:
0
Growth Rate:
0.00%
Total runs:
22
Run Growth:
3
Growth Rate:
13.64%
Total runs:
22
Run Growth:
12
Growth Rate:
54.55%
Total runs:
18
Run Growth:
12
Growth Rate:
66.67%
Total runs:
16
Run Growth:
0
Growth Rate:
0.00%
Total runs:
14
Run Growth:
1
Growth Rate:
7.14%
Total runs:
14
Run Growth:
8
Growth Rate:
57.14%
Total runs:
14
Run Growth:
6
Growth Rate:
42.86%
Total runs:
13
Run Growth:
6
Growth Rate:
46.15%
Total runs:
13
Run Growth:
7
Growth Rate:
53.85%
Total runs:
13
Run Growth:
-124
Growth Rate:
-953.85%
Total runs:
12
Run Growth:
2
Growth Rate:
16.67%
Total runs:
12
Run Growth:
1
Growth Rate:
8.33%
Total runs:
12
Run Growth:
4
Growth Rate:
33.33%
Total runs:
11
Run Growth:
1
Growth Rate:
9.09%
Total runs:
11
Run Growth:
-1
Growth Rate:
-9.09%
Total runs:
11
Run Growth:
3
Growth Rate:
27.27%
Total runs:
11
Run Growth:
1
Growth Rate:
7.69%
Total runs:
10
Run Growth:
1
Growth Rate:
10.00%
Total runs:
10
Run Growth:
1
Growth Rate:
10.00%
Total runs:
9
Run Growth:
2
Growth Rate:
22.22%
Total runs:
8
Run Growth:
0
Growth Rate:
0.00%
Total runs:
8
Run Growth:
-2
Growth Rate:
-28.57%
Total runs:
8
Run Growth:
0
Growth Rate:
0.00%
Total runs:
8
Run Growth:
1
Growth Rate:
12.50%
Total runs:
8
Run Growth:
0
Growth Rate:
0.00%
Total runs:
8
Run Growth:
-1
Growth Rate:
-12.50%
Total runs:
8
Run Growth:
1
Growth Rate:
14.29%
Total runs:
7
Run Growth:
-1
Growth Rate:
-14.29%
Total runs:
7
Run Growth:
-1
Growth Rate:
-14.29%
Total runs:
7
Run Growth:
-30
Growth Rate:
-428.57%
Total runs:
7
Run Growth:
-1
Growth Rate:
-14.29%
Total runs:
6
Run Growth:
-24
Growth Rate:
-400.00%
Total runs:
6
Run Growth:
-7
Growth Rate:
-175.00%
Total runs:
6
Run Growth:
0
Growth Rate:
0.00%
Total runs:
5
Run Growth:
-1
Growth Rate:
-20.00%
Total runs:
4
Run Growth:
-5
Growth Rate:
-125.00%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4
Run Growth:
2
Growth Rate:
50.00%
Total runs:
4
Run Growth:
-1
Growth Rate:
-25.00%
Total runs:
4
Run Growth:
-11
Growth Rate:
-275.00%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3
Run Growth:
1
Growth Rate:
33.33%
Total runs:
3
Run Growth:
2
Growth Rate:
66.67%
Total runs:
3
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3
Run Growth:
2
Growth Rate:
66.67%
Total runs:
3
Run Growth:
-8
Growth Rate:
-266.67%
Total runs:
2
Run Growth:
0
Growth Rate:
0.00%
Total runs:
2
Run Growth:
-1
Growth Rate:
-50.00%
Total runs:
1
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1
Run Growth:
1
Growth Rate:
100.00%
Total runs:
1
Run Growth:
1
Growth Rate:
100.00%
Total runs:
1
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
-1
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%