Total runs:
8.4K
Run Growth:
1.9K
Growth Rate:
22.65%
Sponsored by
APIMart
PKU-Alignment / Beaver-Vision-11B
Model's Last Updated: November 10 2024
image-text-to-text
Introduction of Beaver-Vision-11B
Model Details of Beaver-Vision-11B
🦫 Beaver-Vision-11B
Beaver-Vision-11B is a Image-Text-to-Text chat assistant trained based on the LLaMA-3.2-11B-Vision model using the Align-Anything-Instruct dataset and Align-Anything framework.
Beaver-Vision-11B aims to enhance the instruction-following abilities of MLLMs (Multi-modal Large Language Models). Compared with LLaMA-3.2-11B-Vision-Instruct , Beaver-Vision-11B uses Align-Anything-Instruct dataset and post-training alignment method, achieving better performance.
- Developed by: the PKU-Alignment Team.
- Model Type: An auto-regressive multi-modal (Image-Text-to-Text) language model based on the transformer architecture.
- Fine-tuned from model: LLaMA-3.2-11B-Vision-Instruct .
Model Sources
- Repository: https://github.com/PKU-Alignment/align-anything
- Dataset: https://huggingface.co/datasets/PKU-Alignment/Align-Anything
Evaluation Results
| Metrics | LLaMA-3.2-11B-Vision-Instruct | Beaver-Vision-11B | |
|---|---|---|---|
| LLaVA-Bench-Coco | Rating | 102.36 | 104.67 |
| LLaVA-Bench-in-the-Wild | Rating | 92.76 | 99.13 |
| POPE | Accuracy | 80.71 | 87.07 |
| Recall | 70.95 | 83.02 | |
| Precision | 88.35 | 90.46 | |
| F1 Score | 78.65 | 86.55 |
Example
Comparison
| Role | Content |
|---|---|
| User | What's the name of the restaurant serving these dishes? |
|
|
| LLaMA-3.2-11B-Vision-Instruct | The dish in the image is from a ramen restaurant called "Ippudo." Ippudo is known for its high-quality, authentic Japanese ramen. The bowl with the distinctive design and rich broth is characteristic of their signature Ramen. |
| GPT-4o | The restaurant serving these dishes is ICHIRAN. |
| Beaver-Vision-11B | The image shows that the restaurant is ICHIRAN. Because it appears to be a ramen dish served in a traditional Japanese-style bowl with various accompaniments such as green onions and possibly some kind of meat or broth flavoring. Additionally, the presence of the ICHIRAN logo on the dishware helps identify the restaurant. |
In this example, LLaMA-3.2-11B-Vision-Instruct failed to recognize the logo text in the bowl and fell into hallucination; GPT-4o recognized accurately but did not provide more understandable information, while Beaver-Vision-11B was able to offer both accurate and detailed answers.
Further Examples
|
|
|
|---|---|---|
promptWhat are the things I should be cautious about when I visit here? |
promptWhat is unusual about this image? |
promptWrite an attractive product description for this. |
response
When visiting a place like this, which appears to be a serene lake with a dock and surrounded by forested mountains, you should be cautious of several factors:
|
responseThe unusual aspect of this image is that a person is ironing clothes on the back of a moving taxi. This is an unconventional and unsafe activity, as it involves using an electric iron in close proximity to other vehicles and potentially causing harm or damage. Additionally, the presence of the ironing board and iron on the side of a speeding vehicle defies common sense and safety protocols. |
response
Classic Mario Themed Mugs
|
Usage
import requests
import torch
from PIL import Image
from transformers import MllamaForConditionalGeneration, AutoProcessor
model_id = "PKU-Alignment/Beaver-Vision-11B"
model = MllamaForConditionalGeneration.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
image = Image.open(requests.get(url, stream=True).raw)
messages = [
{"role": "user", "content": [
{"type": "image"},
{"type": "text", "text": "If I had to write a haiku for this one, it would be: "}
]}
]
input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(
image,
input_text,
add_special_tokens=False,
return_tensors="pt"
).to(model.device)
output = model.generate(**inputs, max_new_tokens=30)
print(processor.decode(output[0]))
Citation
Please cite our work if you use the data or model in your paper.
@misc{align_anything,
author = {PKU-Alignment Team},
title = {Align Anything: training all modality models to follow instructions with unified language feedback},
year = {2024},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/PKU-Alignment/align-anything}},
}
License
Beaver-Vision-11B is released under Apache License 2.0, and you also need to agree with LLAMA 3.2 COMMUNITY LICENSE.
Runs of PKU-Alignment Beaver-Vision-11B on huggingface.co
20
Total runs
0
24-hour runs
1
3-day runs
5
7-day runs
10
30-day runs
More Information About Beaver-Vision-11B huggingface.co Model
Url of Beaver-Vision-11B
Beaver-Vision-11B huggingface.co Url
Provider of Beaver-Vision-11B huggingface.co
Other API from PKU-Alignment
Updated:July 10 2023
Total runs:
3.6K
Run Growth:
-2.8K
Growth Rate:
-76.80%
Total runs:
3.0K
Run Growth:
-2.5K
Growth Rate:
-83.35%
Total runs:
1.7K
Run Growth:
-2.1K
Growth Rate:
-120.16%
Total runs:
274
Run Growth:
-144
Growth Rate:
-52.55%
Total runs:
95
Run Growth:
6
Growth Rate:
6.32%
Total runs:
46
Run Growth:
-173
Growth Rate:
-376.09%
Total runs:
37
Run Growth:
14
Growth Rate:
37.84%
Total runs:
36
Run Growth:
17
Growth Rate:
47.22%
Total runs:
33
Run Growth:
-22
Growth Rate:
-66.67%
Total runs:
32
Run Growth:
9
Growth Rate:
28.13%
Total runs:
30
Run Growth:
16
Growth Rate:
57.14%
Total runs:
29
Run Growth:
-163
Growth Rate:
-562.07%
Total runs:
29
Run Growth:
3
Growth Rate:
11.11%
Total runs:
27
Run Growth:
10
Growth Rate:
38.46%
Total runs:
27
Run Growth:
10
Growth Rate:
37.04%
Total runs:
26
Run Growth:
16
Growth Rate:
61.54%
Total runs:
25
Run Growth:
7
Growth Rate:
28.00%
Total runs:
25
Run Growth:
-101
Growth Rate:
-404.00%
Total runs:
25
Run Growth:
5
Growth Rate:
22.73%
Total runs:
24
Run Growth:
12
Growth Rate:
52.17%
Total runs:
24
Run Growth:
8
Growth Rate:
33.33%
Total runs:
24
Run Growth:
24
Growth Rate:
100.00%
Total runs:
24
Run Growth:
2
Growth Rate:
8.70%
Total runs:
23
Run Growth:
7
Growth Rate:
30.43%
Total runs:
23
Run Growth:
8
Growth Rate:
36.36%
Total runs:
23
Run Growth:
9
Growth Rate:
39.13%
Total runs:
23
Run Growth:
4
Growth Rate:
18.18%
Total runs:
21
Run Growth:
6
Growth Rate:
30.00%
Total runs:
20
Run Growth:
13
Growth Rate:
65.00%
Total runs:
20
Run Growth:
-1
Growth Rate:
-5.26%
Total runs:
20
Run Growth:
10
Growth Rate:
52.63%
Total runs:
18
Run Growth:
9
Growth Rate:
50.00%
Total runs:
18
Run Growth:
18
Growth Rate:
100.00%
Total runs:
18
Run Growth:
7
Growth Rate:
41.18%
Total runs:
18
Run Growth:
3
Growth Rate:
17.65%
Total runs:
17
Run Growth:
-7
Growth Rate:
-38.89%
Total runs:
17
Run Growth:
3
Growth Rate:
18.75%
Total runs:
17
Run Growth:
6
Growth Rate:
37.50%
Total runs:
17
Run Growth:
3
Growth Rate:
18.75%
Total runs:
17
Run Growth:
6
Growth Rate:
37.50%
Total runs:
17
Run Growth:
4
Growth Rate:
25.00%
Total runs:
17
Run Growth:
3
Growth Rate:
18.75%
Total runs:
16
Run Growth:
0
Growth Rate:
0.00%
Total runs:
16
Run Growth:
-3
Growth Rate:
-20.00%
Total runs:
16
Run Growth:
-6
Growth Rate:
-50.00%
Total runs:
15
Run Growth:
15
Growth Rate:
100.00%
Total runs:
15
Run Growth:
1
Growth Rate:
7.14%
Total runs:
15
Run Growth:
2
Growth Rate:
14.29%
Total runs:
15
Run Growth:
0
Growth Rate:
0.00%
Total runs:
14
Run Growth:
-1
Growth Rate:
-7.69%
Total runs:
14
Run Growth:
1
Growth Rate:
7.69%
Total runs:
14
Run Growth:
-6
Growth Rate:
-46.15%
Total runs:
13
Run Growth:
-2
Growth Rate:
-16.67%
Total runs:
13
Run Growth:
-8
Growth Rate:
-61.54%
Total runs:
12
Run Growth:
5
Growth Rate:
41.67%
Total runs:
11
Run Growth:
0
Growth Rate:
0.00%
Total runs:
11
Run Growth:
-8
Growth Rate:
-80.00%
Total runs:
10
Run Growth:
0
Growth Rate:
0.00%
Total runs:
10
Run Growth:
7
Growth Rate:
70.00%
Total runs:
10
Run Growth:
0
Growth Rate:
0.00%
Total runs:
9
Run Growth:
9
Growth Rate:
100.00%
Total runs:
9
Run Growth:
4
Growth Rate:
44.44%
Total runs:
9
Run Growth:
0
Growth Rate:
0.00%
Total runs:
9
Run Growth:
0
Growth Rate:
0.00%
Total runs:
8
Run Growth:
7
Growth Rate:
87.50%
Total runs:
8
Run Growth:
8
Growth Rate:
100.00%
Total runs:
8
Run Growth:
0
Growth Rate:
0.00%
Total runs:
8
Run Growth:
0
Growth Rate:
0.00%
Total runs:
7
Run Growth:
6
Growth Rate:
85.71%
Total runs:
7
Run Growth:
7
Growth Rate:
100.00%
Total runs:
7
Run Growth:
6
Growth Rate:
85.71%
Total runs:
7
Run Growth:
6
Growth Rate:
85.71%
Total runs:
6
Run Growth:
6
Growth Rate:
100.00%
Total runs:
6
Run Growth:
6
Growth Rate:
100.00%
Total runs:
6
Run Growth:
6
Growth Rate:
100.00%
Total runs:
6
Run Growth:
3
Growth Rate:
50.00%
Total runs:
6
Run Growth:
5
Growth Rate:
83.33%
Total runs:
6
Run Growth:
3
Growth Rate:
50.00%
Total runs:
6
Run Growth:
6
Growth Rate:
100.00%
Total runs:
6
Run Growth:
6
Growth Rate:
100.00%
Total runs:
6
Run Growth:
6
Growth Rate:
100.00%
Total runs:
6
Run Growth:
6
Growth Rate:
100.00%
Total runs:
6
Run Growth:
0
Growth Rate:
0.00%
Total runs:
6
Run Growth:
6
Growth Rate:
100.00%
Total runs:
6
Run Growth:
6
Growth Rate:
100.00%
Total runs:
6
Run Growth:
3
Growth Rate:
50.00%
Total runs:
6
Run Growth:
5
Growth Rate:
83.33%
Total runs:
6
Run Growth:
0
Growth Rate:
0.00%
Total runs:
6
Run Growth:
0
Growth Rate:
0.00%
Total runs:
6
Run Growth:
6
Growth Rate:
100.00%
Total runs:
6
Run Growth:
6
Growth Rate:
100.00%
Total runs:
5
Run Growth:
0
Growth Rate:
0.00%
Total runs:
5
Run Growth:
0
Growth Rate:
0.00%
Total runs:
5
Run Growth:
2
Growth Rate:
40.00%
Total runs:
5
Run Growth:
5
Growth Rate:
100.00%
Total runs:
5
Run Growth:
5
Growth Rate:
100.00%
Total runs:
5
Run Growth:
5
Growth Rate:
100.00%