Sponsored by
APIMart
ai-forever / ruGPT-3.5-13B
Model's Last Updated: December 05 2023
text-generation
Introduction of ruGPT-3.5-13B
Model Details of ruGPT-3.5-13B
🗿 ruGPT-3.5 13B
Language model for Russian. Model has 13B parameters as you can guess from it's name. This is our biggest model so far and it was used for trainig GigaChat (read more about it in the article ).
Dataset
Model was pretrained on a 300Gb of various domains, than additionaly trained on the 100 Gb of code and legal documets. Here is the dataset structure:
Training data was deduplicated, the text deduplication includes 64-bit hashing of each text in the corpus for keeping texts with a unique hash. We also filter the documents based on their text compression rate using zlib4. The most strongly and weakly compressing deduplicated texts are discarded.
Technical details
Model was trained using Deepspeed and Megatron libraries, on 300B tokens dataset for 3 epochs, around 45 days on 512 V100. After that model was finetuned 1 epoch with sequence length 2048 around 20 days on 200 GPU A100 on additional data (see above).
After the final training perplexity for this model was around 8.8 for Russian.
Examples of usage
Try different generation strategies to reach better results.
request = "Стих про программиста может быть таким:"
encoded_input = tokenizer(request, return_tensors='pt', \
add_special_tokens=False).to('cuda:0')
output = model.generate(
**encoded_input,
num_beams=2,
do_sample=True,
max_new_tokens=100
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
>>> Стих про программиста может быть таким:
Программист сидит в кресле,
Стих сочиняет он про любовь,
Он пишет, пишет, пишет, пишет...
И не выходит ни черта!
request = "Нейронная сеть — это"
encoded_input = tokenizer(request, return_tensors='pt', \
add_special_tokens=False).to('cuda:0')
output = model.generate(
**encoded_input,
num_beams=4,
do_sample=True,
max_new_tokens=100
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
>>> Нейронная сеть — это математическая модель, состоящая из большого
количества нейронов, соединенных между собой электрическими связями.
Нейронная сеть может быть смоделирована на компьютере, и с ее помощью
можно решать задачи, которые не поддаются решению с помощью традиционных
математических методов.
request = "Гагарин полетел в космос в"
encoded_input = tokenizer(request, return_tensors='pt', \
add_special_tokens=False).to('cuda:0')
output = model.generate(
**encoded_input,
num_beams=2,
do_sample=True,
max_new_tokens=100
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
>>> Гагарин полетел в космос в 1961 году. Это было первое в истории
человечества космическое путешествие. Юрий Гагарин совершил его
на космическом корабле Восток-1. Корабль был запущен с космодрома
Байконур.
Runs of ai-forever ruGPT-3.5-13B on huggingface.co
755
Total runs
0
24-hour runs
21
3-day runs
37
7-day runs
163
30-day runs
More Information About ruGPT-3.5-13B huggingface.co Model
Url of ruGPT-3.5-13B
ruGPT-3.5-13B huggingface.co Url
Provider of ruGPT-3.5-13B huggingface.co
Other API from ai-forever
Total runs:
106.3K
Run Growth:
-21.8K
Growth Rate:
-19.37%
Total runs:
79.2K
Run Growth:
-6.5K
Growth Rate:
-7.46%
Total runs:
49.7K
Run Growth:
-6.4K
Growth Rate:
-12.83%
Total runs:
8.2K
Run Growth:
-4.5K
Growth Rate:
-55.45%
Total runs:
7.9K
Run Growth:
-3.1K
Growth Rate:
-38.86%
Total runs:
7.8K
Run Growth:
6.2K
Growth Rate:
75.43%
Total runs:
5.5K
Run Growth:
1.2K
Growth Rate:
20.88%
Total runs:
2.5K
Run Growth:
-1.1K
Growth Rate:
-41.68%
Total runs:
2.4K
Run Growth:
1.3K
Growth Rate:
54.53%
Total runs:
2.1K
Run Growth:
-248
Growth Rate:
-11.55%
Total runs:
1.7K
Run Growth:
-574
Growth Rate:
-34.66%
Total runs:
1.6K
Run Growth:
-382
Growth Rate:
-22.96%
Total runs:
1.3K
Run Growth:
1.3K
Growth Rate:
100.00%
Total runs:
919
Run Growth:
-320
Growth Rate:
-34.56%
Total runs:
898
Run Growth:
323
Growth Rate:
29.80%
Total runs:
780
Run Growth:
460
Growth Rate:
58.97%
Total runs:
676
Run Growth:
-1.4K
Growth Rate:
-214.78%
Total runs:
635
Run Growth:
111
Growth Rate:
16.35%
Total runs:
602
Run Growth:
425
Growth Rate:
70.60%
Total runs:
530
Run Growth:
168
Growth Rate:
32.37%
Total runs:
415
Run Growth:
225
Growth Rate:
52.57%
Total runs:
405
Run Growth:
-73
Growth Rate:
-18.43%
Total runs:
354
Run Growth:
-240
Growth Rate:
-67.23%
Total runs:
339
Run Growth:
108
Growth Rate:
32.24%
Total runs:
328
Run Growth:
-94
Growth Rate:
-28.75%
Total runs:
324
Run Growth:
-14.2K
Growth Rate:
-4336.39%
Total runs:
301
Run Growth:
93
Growth Rate:
30.69%
Total runs:
212
Run Growth:
-101
Growth Rate:
-45.50%
Total runs:
194
Run Growth:
21
Growth Rate:
11.23%
Total runs:
192
Run Growth:
-61
Growth Rate:
-33.70%
Total runs:
180
Run Growth:
-94
Growth Rate:
-51.93%
Total runs:
175
Run Growth:
-21
Growth Rate:
-12.50%
Total runs:
173
Run Growth:
-15
Growth Rate:
-8.67%
Total runs:
165
Run Growth:
-12
Growth Rate:
-7.45%
Total runs:
162
Run Growth:
-17
Growth Rate:
-10.90%
Total runs:
162
Run Growth:
36
Growth Rate:
22.09%
Total runs:
120
Run Growth:
10
Growth Rate:
8.40%
Total runs:
114
Run Growth:
18
Growth Rate:
15.93%
Total runs:
113
Run Growth:
21
Growth Rate:
18.26%
Total runs:
112
Run Growth:
-23
Growth Rate:
-20.35%
Total runs:
111
Run Growth:
12
Growth Rate:
10.81%
Total runs:
110
Run Growth:
20
Growth Rate:
17.70%
Total runs:
101
Run Growth:
10
Growth Rate:
9.71%
Total runs:
100
Run Growth:
12
Growth Rate:
11.88%
Total runs:
98
Run Growth:
-14
Growth Rate:
-14.29%
Total runs:
97
Run Growth:
-23
Growth Rate:
-23.47%
Total runs:
96
Run Growth:
-16
Growth Rate:
-16.67%
Total runs:
95
Run Growth:
4
Growth Rate:
4.17%
Total runs:
94
Run Growth:
2
Growth Rate:
2.11%
Total runs:
94
Run Growth:
-17
Growth Rate:
-17.89%
Total runs:
92
Run Growth:
0
Growth Rate:
0.00%
Total runs:
92
Run Growth:
22
Growth Rate:
23.91%
Total runs:
90
Run Growth:
-128
Growth Rate:
-130.61%
Total runs:
85
Run Growth:
-114
Growth Rate:
-132.56%
Total runs:
85
Run Growth:
-48
Growth Rate:
-57.14%
Total runs:
82
Run Growth:
53
Growth Rate:
64.63%
Total runs:
67
Run Growth:
3
Growth Rate:
4.35%
Total runs:
65
Run Growth:
36
Growth Rate:
55.38%
Total runs:
53
Run Growth:
30
Growth Rate:
56.60%
Total runs:
45
Run Growth:
19
Growth Rate:
42.22%
Total runs:
43
Run Growth:
20
Growth Rate:
46.51%
Total runs:
39
Run Growth:
8
Growth Rate:
20.51%
Total runs:
39
Run Growth:
19
Growth Rate:
48.72%
Total runs:
38
Run Growth:
-8
Growth Rate:
-21.05%
Total runs:
38
Run Growth:
-16
Growth Rate:
-42.11%
Total runs:
38
Run Growth:
8
Growth Rate:
21.05%
Total runs:
36
Run Growth:
0
Growth Rate:
0.00%
Total runs:
32
Run Growth:
-1
Growth Rate:
-3.13%
Total runs:
31
Run Growth:
-4
Growth Rate:
-12.90%
Total runs:
30
Run Growth:
0
Growth Rate:
0.00%
Total runs:
19
Run Growth:
19
Growth Rate:
100.00%
Total runs:
18
Run Growth:
2
Growth Rate:
12.50%
Total runs:
16
Run Growth:
-16
Growth Rate:
-100.00%
Total runs:
12
Run Growth:
-5
Growth Rate:
-41.67%