Total runs:
199.2K
Run Growth:
52.2K
Growth Rate:
26.79%
Sponsored by
APIMart
ai-forever / mGPT
Model's Last Updated: December 05 2023
text-generation
Introduction of mGPT
Model Details of mGPT
Multilingual GPT model
We introduce a family of autoregressive GPT-like models with 1.3 billion parameters trained on 61 languages from 25 language families using Wikipedia and Colossal Clean Crawled Corpus.
We reproduce the GPT-3 architecture using GPT-2 sources and the sparse attention mechanism, Deepspeed and Megatron frameworks allows us to effectively parallelize the training and inference steps. The resulting models show performance on par with the recently released XGLM models at the same time covering more languages and enhancing NLP possibilities for low resource languages.
Code
The source code for the mGPT XL model is available on Github
Paper
mGPT: Few-Shot Learners Go Multilingual
@misc{https://doi.org/10.48550/arxiv.2204.07580,
doi = {10.48550/ARXIV.2204.07580},
url = {https://arxiv.org/abs/2204.07580},
author = {Shliazhko, Oleh and Fenogenova, Alena and Tikhonova, Maria and Mikhailov, Vladislav and Kozlova, Anastasia and Shavrina, Tatiana},
keywords = {Computation and Language (cs.CL), Artificial Intelligence (cs.AI), FOS: Computer and information sciences, FOS: Computer and information sciences, I.2; I.2.7, 68-06, 68-04, 68T50, 68T01},
title = {mGPT: Few-Shot Learners Go Multilingual},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution 4.0 International}
}
Languages
Model supports 61 languages:
ISO codes:
ar he vi id jv ms tl lv lt eu ml ta te hy bn mr hi ur af da en de sv fr it pt ro es el os tg fa ja ka ko th bxr xal mn sw yo be bg ru uk pl my uz ba kk ky tt az cv tr tk tyv sax et fi hu
Languages:
Arabic, Hebrew, Vietnamese, Indonesian, Javanese, Malay, Tagalog, Latvian, Lithuanian, Basque, Malayalam, Tamil, Telugu, Armenian, Bengali, Marathi, Hindi, Urdu, Afrikaans, Danish, English, German, Swedish, French, Italian, Portuguese, Romanian, Spanish, Greek, Ossetian, Tajik, Persian, Japanese, Georgian, Korean, Thai, Buryat, Kalmyk, Mongolian, Swahili, Yoruba, Belarusian, Bulgarian, Russian, Ukrainian, Polish, Burmese, Uzbek, Bashkir, Kazakh, Kyrgyz, Tatar, Azerbaijani, Chuvash, Turkish, Turkmen, Tuvan, Yakut, Estonian, Finnish, Hungarian
Training Data Statistics
- Size: 488 Billion UTF characters
"General training corpus statistics"
Details
The model was trained with sequence length 512 using Megatron and Deepspeed libs by SberDevices team on a dataset of 600 GB of texts in 61 languages. The model has seen 440 billion BPE tokens in total.
Total training time was around 14 days on 256 Nvidia V100 GPUs.
Runs of ai-forever mGPT on huggingface.co
4.7K
Total runs
0
24-hour runs
4
3-day runs
-3.7K
7-day runs
-7.1K
30-day runs
More Information About mGPT huggingface.co Model
Url of mGPT
mGPT huggingface.co Url
Provider of mGPT huggingface.co
Other API from ai-forever
Updated:September 26 2024
Total runs:
156.4K
Run Growth:
134.3K
Growth Rate:
87.68%
Total runs:
35.8K
Run Growth:
-10.2K
Growth Rate:
-28.80%
Total runs:
13.0K
Run Growth:
3.2K
Growth Rate:
24.44%
Total runs:
10.6K
Run Growth:
120
Growth Rate:
1.13%
Total runs:
5.7K
Run Growth:
5.2K
Growth Rate:
92.11%
Total runs:
5.6K
Run Growth:
218
Growth Rate:
3.94%
Total runs:
3.8K
Run Growth:
387
Growth Rate:
10.01%
Total runs:
3.0K
Run Growth:
2.7K
Growth Rate:
92.61%
Total runs:
2.5K
Run Growth:
285
Growth Rate:
11.51%
Total runs:
2.2K
Run Growth:
1.0K
Growth Rate:
43.44%
Total runs:
2.0K
Run Growth:
458
Growth Rate:
22.46%
Total runs:
1.4K
Run Growth:
1.3K
Growth Rate:
94.21%
Total runs:
780
Run Growth:
460
Growth Rate:
58.97%
Total runs:
778
Run Growth:
476
Growth Rate:
61.50%
Total runs:
757
Run Growth:
262
Growth Rate:
35.22%
Total runs:
694
Run Growth:
-1.3K
Growth Rate:
-192.94%
Total runs:
602
Run Growth:
425
Growth Rate:
70.60%
Total runs:
563
Run Growth:
-22
Growth Rate:
-3.99%
Total runs:
460
Run Growth:
11
Growth Rate:
2.22%
Total runs:
438
Run Growth:
-35.9K
Growth Rate:
-2620.35%
Total runs:
373
Run Growth:
240
Growth Rate:
65.04%
Total runs:
334
Run Growth:
10
Growth Rate:
2.97%
Total runs:
331
Run Growth:
241
Growth Rate:
64.78%
Total runs:
310
Run Growth:
288
Growth Rate:
82.29%
Total runs:
278
Run Growth:
165
Growth Rate:
51.40%
Total runs:
275
Run Growth:
-290
Growth Rate:
-103.57%
Total runs:
270
Run Growth:
-10
Growth Rate:
-3.69%
Total runs:
198
Run Growth:
145
Growth Rate:
73.60%
Total runs:
177
Run Growth:
160
Growth Rate:
76.19%
Total runs:
163
Run Growth:
15
Growth Rate:
9.38%
Total runs:
163
Run Growth:
92
Growth Rate:
57.86%
Total runs:
162
Run Growth:
-185
Growth Rate:
-117.83%
Total runs:
159
Run Growth:
106
Growth Rate:
67.95%
Total runs:
157
Run Growth:
146
Growth Rate:
73.37%
Total runs:
151
Run Growth:
-292
Growth Rate:
-185.99%
Total runs:
119
Run Growth:
42
Growth Rate:
34.71%
Total runs:
117
Run Growth:
32
Growth Rate:
26.89%
Total runs:
116
Run Growth:
67
Growth Rate:
56.78%
Total runs:
113
Run Growth:
70
Growth Rate:
60.87%
Total runs:
112
Run Growth:
66
Growth Rate:
58.41%
Total runs:
112
Run Growth:
61
Growth Rate:
55.45%
Total runs:
107
Run Growth:
61
Growth Rate:
54.95%
Total runs:
106
Run Growth:
36
Growth Rate:
34.29%
Total runs:
103
Run Growth:
42
Growth Rate:
41.58%
Total runs:
102
Run Growth:
56
Growth Rate:
54.37%
Total runs:
101
Run Growth:
57
Growth Rate:
55.88%
Total runs:
101
Run Growth:
55
Growth Rate:
55.00%
Total runs:
101
Run Growth:
57
Growth Rate:
55.88%
Total runs:
100
Run Growth:
62
Growth Rate:
59.62%
Total runs:
100
Run Growth:
53
Growth Rate:
51.96%
Total runs:
99
Run Growth:
57
Growth Rate:
57.00%
Total runs:
98
Run Growth:
54
Growth Rate:
54.55%
Total runs:
95
Run Growth:
48
Growth Rate:
50.00%
Total runs:
94
Run Growth:
48
Growth Rate:
51.06%
Total runs:
93
Run Growth:
49
Growth Rate:
52.13%
Total runs:
92
Run Growth:
22
Growth Rate:
23.91%
Total runs:
82
Run Growth:
53
Growth Rate:
64.63%
Total runs:
70
Run Growth:
34
Growth Rate:
47.89%
Total runs:
45
Run Growth:
19
Growth Rate:
42.22%
Total runs:
43
Run Growth:
20
Growth Rate:
46.51%
Total runs:
39
Run Growth:
8
Growth Rate:
20.51%
Total runs:
39
Run Growth:
19
Growth Rate:
48.72%
Total runs:
38
Run Growth:
-8
Growth Rate:
-21.05%
Total runs:
38
Run Growth:
-16
Growth Rate:
-42.11%
Total runs:
38
Run Growth:
6
Growth Rate:
17.14%
Total runs:
38
Run Growth:
8
Growth Rate:
21.05%
Total runs:
36
Run Growth:
0
Growth Rate:
0.00%
Total runs:
32
Run Growth:
5
Growth Rate:
17.86%
Total runs:
32
Run Growth:
-1
Growth Rate:
-3.13%
Total runs:
31
Run Growth:
-4
Growth Rate:
-12.90%
Total runs:
30
Run Growth:
0
Growth Rate:
0.00%
Total runs:
19
Run Growth:
19
Growth Rate:
100.00%
Total runs:
16
Run Growth:
10
Growth Rate:
66.67%
Total runs:
16
Run Growth:
-16
Growth Rate:
-100.00%
Total runs:
10
Run Growth:
2
Growth Rate:
18.18%