Total runs:
31.5K
Run Growth:
-8.5K
Growth Rate:
-26.87%
Sponsored by
iMean AI Trip Planner
FreedomIntelligence / ALLaVA-3B
Model's Last Updated: March 05 2024
text-generation
Introduction of ALLaVA-3B
Model Details of ALLaVA-3B
ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision-Language Model
⚡ALLaVA is a project that provides a large-scale GPT4V-synthesized dataset for training LVLMs.⚡
📃 Paper • 🌐 Demo • 👨🏻💻 Github
🤗 ALLaVA-3B-Longer • 🤗 ALLaVA-3B
Benchmark Result
Our model ALLaVA-3B-Longer and ALLaVA-3B achieve competitive results on 12 benchmarks. Bold numbers denote the SOTA performance among 3B-scale models.
| Model | Backbone | Vicuna-80 | MMB | SEEDBench-v1 (img) | MM-Vet | MMMU (val) | MME | TextVQA | GQA | EMT (CIFAR10) | MLLM-Bench | TouchStone | LLaVA (In-the-Wild) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen-VL-Chat | Qwen-7B | - | 60.6 | 65.4 | - | 35.9 | 1487.5 | 61.5 | 57.5 | - | 6.2 | 711.6 | - |
| LLaVA-v1.5-7B | Vicuna-7B | - | 64.3 | - | 31.1 | - | 1510.7 | 58.2 | 62.0 | - | - | 65.4 | |
| LLaVA-v1.5-13B | Vicuna-13B | 22.50 | 67.7 | 68.2 | 35.4 | 36.4 | 1531.3 | 61.3 | 63.3 | 85.0 | 7.4 | 637.7 | 70.7 |
| ShareGPT4V-7B | Vicuna-7B | - | 68.8 | 69.7 | 37.6 | - | 1943.8 | 60.4 | 63.3 | - | - | - | 72.6 |
| TinyGPT-V | Phi2-2.7B | - | - | - | - | - | - | - | 33.6 | - | - | - | - |
| MobileVLM | MobileLLaMA-2.7B | - | 59.6 | - | - | - | 1288.9 | 47.5 | - | - | - | - | - |
| LLaVA-Phi | Phi2-2.7B | - | 59.8 | - | 28.9 | - | 1335.1 | 48.6 | - | - | - | - | - |
| ALLaVA-3B | Phi2-2.7B | 48.8 | 64.0 | 65.2 | 32.2 | 35.3 | 1623.2 | 49.5 | 48.8 | 90.2 | 6.7 | 632.0 | 69.4 |
| ALLaVA-3B-Longer | Phi2-2.7B | 52.5 | 64.6 | 65.6 | 35.5 | 33.2 | 1564.6 | 50.3 | 50.0 | 85.9 | 8.8 | 636.5 | 71.7 |
The detailed information of each benchmark is shown in Table 4 of our technical report .
🏭 Inference
Load from 🤗 (Recommended)
See the example script .
CLI
See here for CLI code snippet.
🏋️♂️ Training
Data
As shown in the table, ALLaVA-3B uses 1M and 1.5M data for PT. and FT., respectively. ALLaVA-3B-Longer trains one more epoch (i.e. 3M in total) for the FT. stage.
Code
The training code is largely based on LLaVA-v1.5 . We wholeheartedly express our gratitude for their invaluable contributions to open-sourcing LVLMs.
Cost
We train our models on 8*A800 GPUs. ALLaVA-3B-Longer takes 8.3h for PT and 21.3h for FT. ALLaVA-3B takes 8.3h for PT and 10.6h for FT. These two models share the same PT procedure.
Hyperparameters
| Global Batch Size | ZeRO Stage | Optimizer | Max LR | Min LR | Scheduler | Max length | Weight decay |
|---|---|---|---|---|---|---|---|
| 256 (PT) / 128 (FT) | 1 | AdamW | 2e-5 | 2e-6 | CosineAnnealingWarmRestarts | 2048 | 0 |
The LM backbone, projector are trainable, while the vision encoder is kept frozen. The trainabilities of each module are the same for both stages.
📚 ALLaVA-4V Data
The majority part of training data is ALLaVA-4V . See here to prepare it for training.
🙌 Contributors
-
Project Leader: Guiming Hardy Chen
-
Data: Shunian Chen, Junying Chen , Xiangbo Wu
-
Evaluation: Ruifei Zhang
-
Deployment: Xiangbo Wu, Zhiyi Zhang
-
Advising: Zhihong Chen , Benyou Wang
-
Others: Jianquan Li, Xiang Wan
📝 Citation
If you find our data useful, please consider citing our work! We are FreedomIntelligence from Shenzhen Research Institute of Big Data and The Chinese University of Hong Kong, Shenzhen
@article{chen2024allava,
title={ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision-Language Model},
author={Chen, Guiming Hardy and Chen, Shunian and Zhang, Ruifei and Chen, Junying and Wu, Xiangbo and Zhang, Zhiyi and Chen, Zhihong and Li, Jianquan and Wan, Xiang and Wang, Benyou},
journal={arXiv preprint arXiv:2402.11684},
year={2024}
}
Runs of FreedomIntelligence ALLaVA-3B on huggingface.co
57
Total runs
0
24-hour runs
-1
3-day runs
23
7-day runs
39
30-day runs
More Information About ALLaVA-3B huggingface.co Model
Url of ALLaVA-3B
ALLaVA-3B huggingface.co Url
Provider of ALLaVA-3B huggingface.co
Other API from FreedomIntelligence
Updated:January 17 2026
Total runs:
2.1K
Run Growth:
333
Growth Rate:
18.11%
Total runs:
1.4K
Run Growth:
769
Growth Rate:
54.81%
Total runs:
1.4K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.3K
Run Growth:
37
Growth Rate:
2.55%
Total runs:
857
Run Growth:
109
Growth Rate:
12.54%
Total runs:
833
Run Growth:
73
Growth Rate:
8.44%
Total runs:
655
Run Growth:
449
Growth Rate:
69.18%
Total runs:
643
Run Growth:
410
Growth Rate:
63.76%
Total runs:
601
Run Growth:
0
Growth Rate:
0.00%
Total runs:
545
Run Growth:
-501
Growth Rate:
-93.30%
Total runs:
497
Run Growth:
-180
Growth Rate:
-36.07%
Total runs:
486
Run Growth:
146
Growth Rate:
30.17%
Total runs:
375
Run Growth:
4
Growth Rate:
1.07%
Total runs:
328
Run Growth:
70
Growth Rate:
20.23%
Total runs:
307
Run Growth:
177
Growth Rate:
61.67%
Total runs:
304
Run Growth:
-105
Growth Rate:
-30.70%
Total runs:
286
Run Growth:
257
Growth Rate:
93.12%
Total runs:
274
Run Growth:
126
Growth Rate:
45.99%
Total runs:
271
Run Growth:
20
Growth Rate:
7.35%
Total runs:
260
Run Growth:
-17
Growth Rate:
-5.90%
Total runs:
240
Run Growth:
-71
Growth Rate:
-30.08%
Total runs:
206
Run Growth:
146
Growth Rate:
80.22%
Total runs:
202
Run Growth:
161
Growth Rate:
79.70%
Total runs:
165
Run Growth:
57
Growth Rate:
31.67%
Total runs:
163
Run Growth:
79
Growth Rate:
49.38%
Total runs:
161
Run Growth:
114
Growth Rate:
70.81%
Total runs:
155
Run Growth:
38
Growth Rate:
24.52%
Total runs:
155
Run Growth:
-186
Growth Rate:
-119.23%
Total runs:
148
Run Growth:
89
Growth Rate:
60.96%
Total runs:
140
Run Growth:
-22
Growth Rate:
-17.60%
Total runs:
139
Run Growth:
110
Growth Rate:
79.71%
Total runs:
135
Run Growth:
47
Growth Rate:
34.81%
Total runs:
126
Run Growth:
9
Growth Rate:
6.98%
Total runs:
120
Run Growth:
112
Growth Rate:
96.55%
Total runs:
118
Run Growth:
-61
Growth Rate:
-54.95%
Total runs:
115
Run Growth:
0
Growth Rate:
0.00%
Total runs:
113
Run Growth:
-13
Growth Rate:
-12.04%
Total runs:
111
Run Growth:
0
Growth Rate:
0.00%
Total runs:
106
Run Growth:
77
Growth Rate:
88.51%
Total runs:
103
Run Growth:
1
Growth Rate:
0.97%
Total runs:
102
Run Growth:
-131
Growth Rate:
-128.43%
Total runs:
100
Run Growth:
-182
Growth Rate:
-175.00%
Total runs:
98
Run Growth:
-73
Growth Rate:
-78.49%
Total runs:
97
Run Growth:
-214
Growth Rate:
-196.33%
Total runs:
96
Run Growth:
43
Growth Rate:
46.24%
Total runs:
88
Run Growth:
21
Growth Rate:
26.25%
Total runs:
84
Run Growth:
64
Growth Rate:
76.19%
Total runs:
76
Run Growth:
56
Growth Rate:
87.50%
Total runs:
74
Run Growth:
0
Growth Rate:
0.00%
Total runs:
73
Run Growth:
-30
Growth Rate:
-43.48%
Total runs:
69
Run Growth:
15
Growth Rate:
20.55%
Total runs:
69
Run Growth:
54
Growth Rate:
78.26%
Total runs:
68
Run Growth:
-8
Growth Rate:
-10.81%
Total runs:
66
Run Growth:
0
Growth Rate:
0.00%
Total runs:
65
Run Growth:
-103
Growth Rate:
-158.46%
Total runs:
64
Run Growth:
21
Growth Rate:
33.33%
Total runs:
64
Run Growth:
-11
Growth Rate:
-23.91%
Total runs:
62
Run Growth:
-57
Growth Rate:
-87.69%
Total runs:
56
Run Growth:
43
Growth Rate:
76.79%
Total runs:
55
Run Growth:
29
Growth Rate:
44.62%
Total runs:
53
Run Growth:
0
Growth Rate:
0.00%
Total runs:
50
Run Growth:
-858
Growth Rate:
-1716.00%
Total runs:
49
Run Growth:
-1
Growth Rate:
-2.04%
Total runs:
46
Run Growth:
-95
Growth Rate:
-206.52%
Total runs:
44
Run Growth:
44
Growth Rate:
100.00%
Total runs:
43
Run Growth:
15
Growth Rate:
35.71%
Total runs:
42
Run Growth:
29
Growth Rate:
63.04%
Total runs:
42
Run Growth:
14
Growth Rate:
33.33%
Total runs:
41
Run Growth:
10
Growth Rate:
23.26%
Total runs:
40
Run Growth:
-32
Growth Rate:
-78.05%
Total runs:
40
Run Growth:
8
Growth Rate:
17.78%
Total runs:
38
Run Growth:
11
Growth Rate:
32.35%
Total runs:
35
Run Growth:
11
Growth Rate:
28.21%
Total runs:
35
Run Growth:
0
Growth Rate:
0.00%
Total runs:
35
Run Growth:
0
Growth Rate:
0.00%
Total runs:
33
Run Growth:
16
Growth Rate:
53.33%
Total runs:
33
Run Growth:
19
Growth Rate:
57.58%
Total runs:
31
Run Growth:
8
Growth Rate:
23.53%
Total runs:
30
Run Growth:
25
Growth Rate:
83.33%
Total runs:
30
Run Growth:
14
Growth Rate:
43.75%
Total runs:
29
Run Growth:
-24
Growth Rate:
-85.71%
Total runs:
28
Run Growth:
13
Growth Rate:
41.94%
Total runs:
26
Run Growth:
0
Growth Rate:
0.00%
Total runs:
25
Run Growth:
8
Growth Rate:
33.33%
Total runs:
24
Run Growth:
14
Growth Rate:
58.33%
Total runs:
24
Run Growth:
13
Growth Rate:
54.17%
Total runs:
23
Run Growth:
14
Growth Rate:
60.87%
Total runs:
22
Run Growth:
12
Growth Rate:
54.55%
Total runs:
22
Run Growth:
16
Growth Rate:
64.00%
Total runs:
22
Run Growth:
9
Growth Rate:
37.50%
Total runs:
21
Run Growth:
0
Growth Rate:
0.00%
Total runs:
21
Run Growth:
13
Growth Rate:
56.52%
Total runs:
20
Run Growth:
12
Growth Rate:
54.55%
Total runs:
19
Run Growth:
9
Growth Rate:
47.37%
Total runs:
18
Run Growth:
4
Growth Rate:
22.22%
Total runs:
17
Run Growth:
2
Growth Rate:
11.11%
Total runs:
16
Run Growth:
2
Growth Rate:
13.33%
Total runs:
15
Run Growth:
7
Growth Rate:
43.75%
Total runs:
15
Run Growth:
9
Growth Rate:
60.00%