Sponsored by
ThumbnailCreator.com
chargoddard / llama33b-s2a4-qlora
Model's Last Updated: September 21 2023
text-generation
Introduction of llama33b-s2a4-qlora
Model Details of llama33b-s2a4-qlora
LLaMA 33b finetuned on
wikitext_document_level
with a combination of both linear and NTK-aware ROPE scaling.
Trained with alpha=4, scale=2. Definitely works for sequence lengths up to and including 4096. Might work for much longer, but I don't have the VRAM to test properly. ¯\_(ツ)_/¯
Training procedure
The following
bitsandbytes
quantization config was used during training:
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: bfloat16
Framework versions
- PEFT 0.4.0.dev0
Runs of chargoddard llama33b-s2a4-qlora on huggingface.co
16
Total runs
-1
24-hour runs
1
3-day runs
0
7-day runs
10
30-day runs
More Information About llama33b-s2a4-qlora huggingface.co Model
Url of llama33b-s2a4-qlora
llama33b-s2a4-qlora huggingface.co Url
Provider of llama33b-s2a4-qlora huggingface.co
Other API from chargoddard
Total runs:
1.0K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
877
Run Growth:
-1.5K
Growth Rate:
-174.46%
Total runs:
771
Run Growth:
380
Growth Rate:
49.29%
Total runs:
757
Run Growth:
-68
Growth Rate:
-8.98%
Total runs:
737
Run Growth:
-106
Growth Rate:
-14.38%
Total runs:
645
Run Growth:
-269
Growth Rate:
-41.71%
Total runs:
620
Run Growth:
-137
Growth Rate:
-22.10%
Total runs:
618
Run Growth:
-155
Growth Rate:
-25.08%
Total runs:
548
Run Growth:
-168
Growth Rate:
-30.66%
Total runs:
391
Run Growth:
47
Growth Rate:
12.02%
Total runs:
216
Run Growth:
-185
Growth Rate:
-85.65%
Total runs:
188
Run Growth:
-339
Growth Rate:
-180.32%
Total runs:
184
Run Growth:
-329
Growth Rate:
-178.80%
Total runs:
183
Run Growth:
-321
Growth Rate:
-175.41%
Total runs:
144
Run Growth:
0
Growth Rate:
0.00%
Total runs:
133
Run Growth:
-100
Growth Rate:
-75.19%
Total runs:
111
Run Growth:
0
Growth Rate:
0.00%
Total runs:
103
Run Growth:
-358
Growth Rate:
-347.57%
Total runs:
84
Run Growth:
-53
Growth Rate:
-66.25%
Total runs:
74
Run Growth:
4
Growth Rate:
5.41%
Total runs:
73
Run Growth:
5
Growth Rate:
6.85%
Total runs:
72
Run Growth:
5
Growth Rate:
6.67%
Total runs:
66
Run Growth:
-69
Growth Rate:
-102.99%
Total runs:
64
Run Growth:
-8
Growth Rate:
-12.50%
Total runs:
63
Run Growth:
-25
Growth Rate:
-39.68%
Total runs:
58
Run Growth:
-35
Growth Rate:
-59.32%
Total runs:
40
Run Growth:
23
Growth Rate:
57.50%
Total runs:
37
Run Growth:
15
Growth Rate:
42.86%
Total runs:
35
Run Growth:
25
Growth Rate:
71.43%
Total runs:
31
Run Growth:
11
Growth Rate:
35.48%
Total runs:
23
Run Growth:
0
Growth Rate:
0.00%
Total runs:
23
Run Growth:
0
Growth Rate:
0.00%
Total runs:
23
Run Growth:
2
Growth Rate:
8.70%
Total runs:
21
Run Growth:
7
Growth Rate:
33.33%
Total runs:
19
Run Growth:
0
Growth Rate:
0.00%
Total runs:
17
Run Growth:
5
Growth Rate:
29.41%
Total runs:
16
Run Growth:
0
Growth Rate:
0.00%
Total runs:
15
Run Growth:
9
Growth Rate:
60.00%
Total runs:
15
Run Growth:
-1
Growth Rate:
-6.67%
Total runs:
14
Run Growth:
3
Growth Rate:
21.43%
Total runs:
13
Run Growth:
0
Growth Rate:
0.00%
Total runs:
11
Run Growth:
1
Growth Rate:
9.09%
Total runs:
9
Run Growth:
5
Growth Rate:
55.56%
Total runs:
7
Run Growth:
-1
Growth Rate:
-14.29%
Total runs:
6
Run Growth:
0
Growth Rate:
0.00%
Total runs:
5
Run Growth:
0
Growth Rate:
0.00%
Total runs:
5
Run Growth:
4
Growth Rate:
80.00%
Total runs:
5
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3
Run Growth:
2
Growth Rate:
66.67%
Total runs:
2
Run Growth:
2
Growth Rate:
100.00%
Total runs:
2
Run Growth:
0
Growth Rate:
0.00%
Total runs:
2
Run Growth:
2
Growth Rate:
100.00%
Total runs:
2
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1
Run Growth:
1
Growth Rate:
100.00%
Total runs:
1
Run Growth:
-2
Growth Rate:
-200.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%