Total runs:
408
Run Growth:
365
Growth Rate:
89.46%
Sponsored by
AdsCreator.com
athirdpath / Nethena-20b-Glue-LORA
Model's Last Updated: November 13 2023
text-generation
Introduction of Nethena-20b-Glue-LORA
Model Details of Nethena-20b-Glue-LORA
This model is a fine-tuned version of NeverSleep/Nethena-20B on a private dataset. It achieves the following results on the evaluation set:
- Loss: 1.3864
Model description
athirdpath/Nethena-20b-Glued-LORA is a 128 rank LORA for RP, trained on NeverSleep/Nethena-20B . It is unalligned and NSFW-oriented.
This is a test, exploring the effects of "gluing" the components of the 20b model together to reduce the iconic word replacement errors, increase lucidity, and improve recall.
Training and evaluation data
The private ~500k token dataset used to train the LORA was Alpaca formatted and focused on 4 primary categories:
- Medical texts (on psychology, reproductive organs, anatomy, and pregnancy). These are formatted so the model, in character as a doctor or therapist, answers a patient's question in short to medium form.
- Excerpts from short stories and novellas (erotic and romantic) centered around both realistic and fantastic situations, covering several fetishes as well. These are sliced into ~2048 token chunks, and these long-form responses are all tied to the command “Enter narrator mode.” in the instructions.
- A selection from PIPPA, using a wide keyword search for tokens associated with low quality human or AI data to remove those responses, then a positive search was done for words and phrases associated with a higher reading level. These are converted to Alpaca with “Enter RP mode.” in all the instruction fields.
- ~18k tokens of GPT-4 generated data on role-playing from various characters’ perspectives, focusing on different situations and emotions. Includes many multi-turn conversations.
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- gradient_accumulation_steps: 5
- total_train_batch_size: 20
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 3
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.955 | 0.38 | 25 | 1.9037 |
| 1.6598 | 0.75 | 50 | 1.6192 |
| 1.5649 | 1.13 | 75 | 1.5010 |
| 1.4424 | 1.5 | 100 | 1.4424 |
| 1.4142 | 1.88 | 125 | 1.4068 |
| 1.4951 | 2.25 | 150 | 1.3908 |
| 1.4418 | 2.63 | 175 | 1.3864 |
Framework versions
- Transformers 4.34.1
- Pytorch 2.1.0+cu118
- Datasets 2.14.6
- Tokenizers 0.14.1
Runs of athirdpath Nethena-20b-Glue-LORA on huggingface.co
16
Total runs
0
24-hour runs
-2
3-day runs
-3
7-day runs
3
30-day runs
More Information About Nethena-20b-Glue-LORA huggingface.co Model
Url of Nethena-20b-Glue-LORA
Nethena-20b-Glue-LORA huggingface.co Url
Provider of Nethena-20b-Glue-LORA huggingface.co
Other API from athirdpath
Updated:May 04 2024
Total runs:
246
Run Growth:
206
Growth Rate:
83.74%
Total runs:
188
Run Growth:
63
Growth Rate:
33.51%
Total runs:
79
Run Growth:
-34
Growth Rate:
-43.04%
Total runs:
75
Run Growth:
-48
Growth Rate:
-64.00%
Total runs:
68
Run Growth:
-53
Growth Rate:
-77.94%
Total runs:
63
Run Growth:
4
Growth Rate:
6.35%
Total runs:
63
Run Growth:
-58
Growth Rate:
-92.06%
Total runs:
62
Run Growth:
-6
Growth Rate:
-9.68%
Total runs:
57
Run Growth:
-65
Growth Rate:
-114.04%
Total runs:
53
Run Growth:
-75
Growth Rate:
-141.51%
Total runs:
51
Run Growth:
-62
Growth Rate:
-121.57%
Total runs:
49
Run Growth:
16
Growth Rate:
32.65%
Total runs:
47
Run Growth:
-67
Growth Rate:
-142.55%
Total runs:
41
Run Growth:
-7
Growth Rate:
-17.07%
Total runs:
38
Run Growth:
-48
Growth Rate:
-126.32%
Total runs:
32
Run Growth:
11
Growth Rate:
34.38%
Total runs:
26
Run Growth:
1
Growth Rate:
3.85%
Total runs:
26
Run Growth:
12
Growth Rate:
46.15%
Total runs:
26
Run Growth:
9
Growth Rate:
34.62%
Total runs:
26
Run Growth:
9
Growth Rate:
34.62%
Total runs:
25
Run Growth:
-52
Growth Rate:
-208.00%
Total runs:
25
Run Growth:
-12
Growth Rate:
-48.00%
Total runs:
25
Run Growth:
9
Growth Rate:
36.00%
Total runs:
25
Run Growth:
3
Growth Rate:
12.00%
Total runs:
22
Run Growth:
7
Growth Rate:
31.82%
Total runs:
22
Run Growth:
2
Growth Rate:
9.09%
Total runs:
20
Run Growth:
8
Growth Rate:
40.00%
Total runs:
20
Run Growth:
7
Growth Rate:
35.00%
Total runs:
20
Run Growth:
9
Growth Rate:
45.00%
Total runs:
20
Run Growth:
13
Growth Rate:
65.00%
Total runs:
19
Run Growth:
10
Growth Rate:
52.63%
Total runs:
19
Run Growth:
9
Growth Rate:
47.37%
Total runs:
19
Run Growth:
14
Growth Rate:
73.68%
Total runs:
19
Run Growth:
11
Growth Rate:
57.89%
Total runs:
19
Run Growth:
8
Growth Rate:
42.11%
Total runs:
18
Run Growth:
7
Growth Rate:
38.89%
Total runs:
18
Run Growth:
9
Growth Rate:
50.00%
Total runs:
18
Run Growth:
-7
Growth Rate:
-38.89%
Total runs:
18
Run Growth:
3
Growth Rate:
16.67%
Total runs:
18
Run Growth:
2
Growth Rate:
11.11%
Total runs:
18
Run Growth:
9
Growth Rate:
50.00%
Total runs:
17
Run Growth:
3
Growth Rate:
17.65%
Total runs:
17
Run Growth:
2
Growth Rate:
11.76%
Total runs:
17
Run Growth:
5
Growth Rate:
29.41%
Total runs:
17
Run Growth:
5
Growth Rate:
29.41%
Total runs:
16
Run Growth:
3
Growth Rate:
18.75%
Total runs:
16
Run Growth:
8
Growth Rate:
50.00%
Total runs:
16
Run Growth:
8
Growth Rate:
50.00%
Total runs:
16
Run Growth:
1
Growth Rate:
6.25%
Total runs:
16
Run Growth:
7
Growth Rate:
43.75%
Total runs:
16
Run Growth:
0
Growth Rate:
0.00%
Total runs:
16
Run Growth:
8
Growth Rate:
50.00%
Total runs:
15
Run Growth:
7
Growth Rate:
46.67%
Total runs:
15
Run Growth:
0
Growth Rate:
0.00%
Total runs:
15
Run Growth:
4
Growth Rate:
26.67%
Total runs:
15
Run Growth:
3
Growth Rate:
20.00%
Total runs:
14
Run Growth:
-16
Growth Rate:
-114.29%
Total runs:
14
Run Growth:
10
Growth Rate:
71.43%
Total runs:
14
Run Growth:
5
Growth Rate:
35.71%
Total runs:
14
Run Growth:
6
Growth Rate:
42.86%
Total runs:
13
Run Growth:
1
Growth Rate:
7.69%
Total runs:
13
Run Growth:
-4
Growth Rate:
-30.77%
Total runs:
13
Run Growth:
3
Growth Rate:
23.08%
Total runs:
13
Run Growth:
4
Growth Rate:
30.77%
Total runs:
13
Run Growth:
0
Growth Rate:
0.00%
Total runs:
13
Run Growth:
6
Growth Rate:
46.15%
Total runs:
13
Run Growth:
-15
Growth Rate:
-115.38%
Total runs:
13
Run Growth:
2
Growth Rate:
15.38%
Total runs:
12
Run Growth:
-1
Growth Rate:
-8.33%
Total runs:
12
Run Growth:
5
Growth Rate:
41.67%
Total runs:
12
Run Growth:
7
Growth Rate:
58.33%
Total runs:
11
Run Growth:
4
Growth Rate:
36.36%
Total runs:
11
Run Growth:
1
Growth Rate:
9.09%
Total runs:
11
Run Growth:
3
Growth Rate:
27.27%
Total runs:
11
Run Growth:
3
Growth Rate:
27.27%
Total runs:
10
Run Growth:
1
Growth Rate:
10.00%
Total runs:
9
Run Growth:
1
Growth Rate:
11.11%
Total runs:
6
Run Growth:
0
Growth Rate:
0.00%