Total runs:
31.5K
Run Growth:
-8.5K
Growth Rate:
-26.87%
Sponsored by
Vmake Labs Video Enhancer
FreedomIntelligence / ALLaVA-Phi3-mini-128k
Model's Last Updated: June 25 2024
text-generation
Introduction of ALLaVA-Phi3-mini-128k
Model Details of ALLaVA-Phi3-mini-128k
ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision-Language Model
⚡ALLaVA is a project that provides a large-scale GPT4V-synthesized dataset for training LVLMs.⚡
📃 Paper • 🌐 Demo • 👨🏻💻 Github
🤗 ALLaVA-Phi3-mini-128k • 🤗 ALLaVA-StableLM2-1_6B • 🤗 ALLaVA-Phi2-2_7B
Benchmark Result
Our models ALLaVA-Phi3-mini-128k , ALLaVA-StableLM2-1_6B and ALLaVA-Phi2-2_7B achieve competitive results on 17 benchmarks.
| Models | Vicuna-80 | GQA | HallusionBench | MME-P | MMVP | TouchStone | TextVQA | MME-C | MathVista | MM-Vet | MMMU-val | SQA (img) | LLaVA (In-the-Wild) | MLLM-Bench | MMB-en | MMB-cn | SEEDBench (img, v1) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Large VLMs | |||||||||||||||||
| BLIP-2 | - | - | - | - | - | - | - | - | - | 22.4 | 34.4 | - | - | 3.0* | - | - | 49.7 |
| InstructBLIP | - | 49.5 | - | - | - | - | - | - | - | 25.6 | - | - | 58.2 | - | 44.0 | - | - |
| Qwen-VL-Chat | - | 57.5 | - | 1487.6 | - | - | 61.5 | 360.7 | - | 31.1 | - | 68.2 | - | - | 60.6 | 56.7 | 65.4 |
| LLaVA-1.5-7B | 13.8* | 62.0 | 36.6* | 1504.4* | 24.7* | 594.9* | 58.2 | 324.6* | 25.0* | 31.1 | 35.1* | 66.8 | 65.4 | 23.0* | 64.3 | 58.3 | 66.1 |
| LLaVA-1.5-13B | 22.5 | 63.3 | 36.5* | 1531.3 | 38.0* | 617.7* | 61.3 | 295.4 | 28.3* | 35.4 | 34.4* | 71.6 | 72.5 | - | 67.7 | 63.6 | 68.2 |
| LVIS-7B | - | 62.6 | - | - | - | - | 58.7 | - | - | 31.5 | - | - | 67.0 | 29.0* | 66.2 | - | - |
| LVIS-13B | - | 63.6* | - | - | - | - | 62.5* | - | - | 37.4* | - | - | 71.3* | - | 68.0* | - | - |
| ShareGPT4V-7B | 13.8* | 63.3 | 36.0* | 1540.1* | 34.0* | 637.2* | 60.4 | 346.1* | 24.7* | 37.6 | 35.4* | 68.4* | 72.6 | 30.2* | 68.8 | 61.0* | 69.7 |
| ShareGPT4V-13B | 17.5* | 64.8 | 39.0* | 1576.1* | 35.3* | 648.7* | 62.2 | 309.3* | 28.8* | 43.1 | 35.6* | 70.0* | 79.9 | 35.5* | 71.2 | 61.7* | 70.8 |
| 4B-scale Lite VLMs | |||||||||||||||||
| MobileVLM-v2 | 5.0* | 61.1 | 30.8* | 1440.5 | 18.7* | 541.0* | 57.5 | 261.8* | 28.3* | 26.1* | 30.8* | 70.0 | 53.2* | 15.7* | 63.2 | 43.2* | 64.5* |
| Mipha-3B | 16.2* | 63.9 | 34.3* | 1488.9 | 32.0* | 619.0* | 56.6 | 285.0* | 27.8* | 33.5* | 35.8* | 70.9 | 64.7* | 23.1* | 69.7 | 42.9* | 71.2 * |
| TinyLLaVA | 15.6* | 62.1 | 37.2* | 1465.5* | 33.3* | 663.5* | 60.3 | 281.1* | 30.3* | 37.5 | 38.4 | 73.0 | 70.8* | 29.8* | 69.7 * | 42.8* | 70.4* |
| Ours | |||||||||||||||||
| ALLaVA-Phi2 | 49.4 | 48.8 | 24.8 | 1316.2 | 36.0 | 632.0 | 49.5 | 301.8 | 27.4 | 32.2 | 35.3 | 67.6 | 69.4 | 43.6 | 64.0 | 40.8 | 65.2 |
| ALLaVA-StableLM2 | 38.8 | 49.8 | 25.3 | 1311.7 | 34.0 | 655.2 | 51.7 | 257.9 | 27.7 | 31.7 | 33.3 | 64.7 | 72.0 | 39.3 | 64.6 | 49.8 | 65.7 |
| ALLaVA-Phi3 | 56.9 | 52.2 | 48.1 | 1382.3 | 32.7 | 667.8 | 53.0 | 347.1 | 32.9 | 37.8 | 41.1 | 64.0 | 68.5 | 54.8 | 68.1 | 55.3 | 69.0 |
* denotes the results of our evaluation. Bold numbers are the best results among all 4B-scale LVLMs.The detailed information of each benchmark is shown in Table 4 of our technical report .
🏭 Inference
All models can be loaded from 🤗 with
.from_pretrained()
.
Check out the
example scripts
and make sure you have the same outputs as shown in the scripts.
🏋️♂️ Training
Data
ALLaVA uses 1.0M and 1.5M data for PT. and FT., respectively.
Code
The training code is largely based on LLaVA-v1.5 . We wholeheartedly express our gratitude for their invaluable contributions to open-sourcing LVLMs.
Hyperparameters
| Global Batch Size | ZeRO Stage | Optimizer | Max LR | Min LR | Scheduler | Weight decay |
|---|---|---|---|---|---|---|
| 256 (PT) / 128 (FT) | 1 | AdamW | 2e-5 | 2e-6 | CosineAnnealingWarmRestarts | 0 |
The LM backbone, projector are trainable, while the vision encoder is kept frozen. The trainabilities of each module are the same for both stages.
📚 ALLaVA-4V Data
The majority part of training data is ALLaVA-4V . See here to prepare it for training.
🙌 Contributors
-
Project Leader: Guiming Hardy Chen
-
Data: Shunian Chen, Junying Chen , Xiangbo Wu
-
Evaluation: Ruifei Zhang
-
Deployment: Xiangbo Wu, Zhiyi Zhang
-
Advising: Zhihong Chen , Benyou Wang
-
Others: Jianquan Li, Xiang Wan
📝 Citation
If you find our data useful, please consider citing our work! We are FreedomIntelligence from Shenzhen Research Institute of Big Data and The Chinese University of Hong Kong, Shenzhen
@article{chen2024allava,
title={ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision-Language Model},
author={Chen, Guiming Hardy and Chen, Shunian and Zhang, Ruifei and Chen, Junying and Wu, Xiangbo and Zhang, Zhiyi and Chen, Zhihong and Li, Jianquan and Wan, Xiang and Wang, Benyou},
journal={arXiv preprint arXiv:2402.11684},
year={2024}
}
Runs of FreedomIntelligence ALLaVA-Phi3-mini-128k on huggingface.co
53
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs
More Information About ALLaVA-Phi3-mini-128k huggingface.co Model
Url of ALLaVA-Phi3-mini-128k
ALLaVA-Phi3-mini-128k huggingface.co Url
Provider of ALLaVA-Phi3-mini-128k huggingface.co
Other API from FreedomIntelligence
Updated:January 17 2026
Total runs:
2.8K
Run Growth:
2.0K
Growth Rate:
68.76%
Total runs:
1.5K
Run Growth:
287
Growth Rate:
19.59%
Total runs:
1.4K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.4K
Run Growth:
-15
Growth Rate:
-1.10%
Total runs:
788
Run Growth:
154
Growth Rate:
19.54%
Total runs:
785
Run Growth:
113
Growth Rate:
14.39%
Total runs:
756
Run Growth:
-34
Growth Rate:
-4.50%
Total runs:
717
Run Growth:
517
Growth Rate:
72.11%
Total runs:
713
Run Growth:
186
Growth Rate:
26.09%
Total runs:
643
Run Growth:
410
Growth Rate:
63.76%
Total runs:
601
Run Growth:
0
Growth Rate:
0.00%
Total runs:
399
Run Growth:
62
Growth Rate:
15.27%
Total runs:
375
Run Growth:
4
Growth Rate:
1.07%
Total runs:
336
Run Growth:
65
Growth Rate:
20.38%
Total runs:
318
Run Growth:
299
Growth Rate:
94.03%
Total runs:
278
Run Growth:
261
Growth Rate:
96.31%
Total runs:
274
Run Growth:
126
Growth Rate:
45.99%
Total runs:
270
Run Growth:
140
Growth Rate:
51.85%
Total runs:
257
Run Growth:
67
Growth Rate:
26.17%
Total runs:
245
Run Growth:
60
Growth Rate:
24.49%
Total runs:
240
Run Growth:
234
Growth Rate:
97.50%
Total runs:
227
Run Growth:
-153
Growth Rate:
-66.23%
Total runs:
218
Run Growth:
-258
Growth Rate:
-118.35%
Total runs:
202
Run Growth:
161
Growth Rate:
79.70%
Total runs:
196
Run Growth:
60
Growth Rate:
30.61%
Total runs:
189
Run Growth:
91
Growth Rate:
48.92%
Total runs:
172
Run Growth:
102
Growth Rate:
59.30%
Total runs:
172
Run Growth:
9
Growth Rate:
5.26%
Total runs:
170
Run Growth:
119
Growth Rate:
70.83%
Total runs:
169
Run Growth:
84
Growth Rate:
50.30%
Total runs:
161
Run Growth:
-301
Growth Rate:
-186.96%
Total runs:
161
Run Growth:
114
Growth Rate:
70.81%
Total runs:
152
Run Growth:
-242
Growth Rate:
-159.21%
Total runs:
134
Run Growth:
110
Growth Rate:
82.09%
Total runs:
132
Run Growth:
55
Growth Rate:
41.67%
Total runs:
120
Run Growth:
-11
Growth Rate:
-8.94%
Total runs:
115
Run Growth:
0
Growth Rate:
0.00%
Total runs:
111
Run Growth:
0
Growth Rate:
0.00%
Total runs:
103
Run Growth:
1
Growth Rate:
0.97%
Total runs:
102
Run Growth:
79
Growth Rate:
77.45%
Total runs:
101
Run Growth:
-79
Growth Rate:
-78.22%
Total runs:
98
Run Growth:
-13
Growth Rate:
-12.50%
Total runs:
96
Run Growth:
26
Growth Rate:
27.08%
Total runs:
91
Run Growth:
-29
Growth Rate:
-30.53%
Total runs:
88
Run Growth:
44
Growth Rate:
50.00%
Total runs:
84
Run Growth:
-70
Growth Rate:
-81.40%
Total runs:
83
Run Growth:
51
Growth Rate:
61.45%
Total runs:
81
Run Growth:
47
Growth Rate:
58.75%
Total runs:
74
Run Growth:
0
Growth Rate:
0.00%
Total runs:
73
Run Growth:
-207
Growth Rate:
-287.50%
Total runs:
68
Run Growth:
-118
Growth Rate:
-173.53%
Total runs:
66
Run Growth:
0
Growth Rate:
0.00%
Total runs:
62
Run Growth:
24
Growth Rate:
38.71%
Total runs:
61
Run Growth:
-24
Growth Rate:
-40.68%
Total runs:
61
Run Growth:
32
Growth Rate:
53.33%
Total runs:
60
Run Growth:
13
Growth Rate:
20.97%
Total runs:
57
Run Growth:
-3
Growth Rate:
-5.26%
Total runs:
56
Run Growth:
43
Growth Rate:
76.79%
Total runs:
50
Run Growth:
-858
Growth Rate:
-1716.00%
Total runs:
50
Run Growth:
-143
Growth Rate:
-286.00%
Total runs:
50
Run Growth:
27
Growth Rate:
54.00%
Total runs:
48
Run Growth:
-47
Growth Rate:
-97.92%
Total runs:
48
Run Growth:
13
Growth Rate:
26.53%
Total runs:
45
Run Growth:
-7
Growth Rate:
-15.56%
Total runs:
44
Run Growth:
44
Growth Rate:
100.00%
Total runs:
43
Run Growth:
-26
Growth Rate:
-57.78%
Total runs:
43
Run Growth:
-35
Growth Rate:
-81.40%
Total runs:
42
Run Growth:
30
Growth Rate:
71.43%
Total runs:
41
Run Growth:
-24
Growth Rate:
-58.54%
Total runs:
41
Run Growth:
8
Growth Rate:
19.51%
Total runs:
39
Run Growth:
1
Growth Rate:
2.56%
Total runs:
38
Run Growth:
-1
Growth Rate:
-2.56%
Total runs:
36
Run Growth:
-3
Growth Rate:
-8.33%
Total runs:
35
Run Growth:
0
Growth Rate:
0.00%
Total runs:
34
Run Growth:
-1
Growth Rate:
-2.94%
Total runs:
32
Run Growth:
4
Growth Rate:
12.12%
Total runs:
30
Run Growth:
25
Growth Rate:
83.33%
Total runs:
30
Run Growth:
8
Growth Rate:
26.67%
Total runs:
29
Run Growth:
-10
Growth Rate:
-34.48%
Total runs:
29
Run Growth:
17
Growth Rate:
58.62%
Total runs:
28
Run Growth:
15
Growth Rate:
55.56%
Total runs:
27
Run Growth:
13
Growth Rate:
48.15%
Total runs:
26
Run Growth:
16
Growth Rate:
61.54%
Total runs:
26
Run Growth:
0
Growth Rate:
0.00%
Total runs:
25
Run Growth:
13
Growth Rate:
52.00%
Total runs:
25
Run Growth:
-5
Growth Rate:
-19.23%
Total runs:
24
Run Growth:
11
Growth Rate:
45.83%
Total runs:
23
Run Growth:
11
Growth Rate:
47.83%
Total runs:
21
Run Growth:
10
Growth Rate:
45.45%
Total runs:
21
Run Growth:
7
Growth Rate:
33.33%
Total runs:
21
Run Growth:
7
Growth Rate:
33.33%
Total runs:
21
Run Growth:
2
Growth Rate:
9.52%
Total runs:
20
Run Growth:
7
Growth Rate:
33.33%
Total runs:
18
Run Growth:
9
Growth Rate:
50.00%
Total runs:
18
Run Growth:
4
Growth Rate:
22.22%
Total runs:
18
Run Growth:
-8
Growth Rate:
-44.44%
Total runs:
18
Run Growth:
-12
Growth Rate:
-66.67%
Total runs:
18
Run Growth:
6
Growth Rate:
33.33%