Total runs:
100.8K
Run Growth:
61.8K
Growth Rate:
61.30%
Sponsored by
Zawa
TIGER-Lab / Mantis-llava-7b
Model's Last Updated: May 18 2024
image-text-to-text
Introduction of Mantis-llava-7b
Model Details of Mantis-llava-7b
Mantis: Interleaved Multi-Image Instruction Tuning (Deprecated)
Mantis is a multimodal conversational AI model that can chat with users about images and text. It's optimized for multi-image reasoning, where interleaved text and images can be used to generate responses.
Note that this is an older version of Mantis , please refer to our newest version at mantis-Siglip-llama3 . The newer version improves significantly over both multi-image and single-image tasks.
Mantis is trained on the newly curated dataset Mantis-Instruct , a large-scale multi-image QA dataset that covers various multi-image reasoning tasks.
Inference
You can install Mantis's GitHub codes as a Python package
pip install git+https://github.com/TIGER-AI-Lab/Mantis.git
then run inference with codes here: examples/run_mantis.py
from mantis.models.mllava import chat_mllava
from PIL import Image
import torch
image1 = "image1.jpg"
image2 = "image2.jpg"
images = [Image.open(image1), Image.open(image2)]
# load processor and model
from mantis.models.mllava import MLlavaProcessor, LlavaForConditionalGeneration
processor = MLlavaProcessor.from_pretrained("TIGER-Lab/Mantis-bakllava-7b")
model = LlavaForConditionalGeneration.from_pretrained("TIGER-Lab/Mantis-bakllava-7b", device_map="auto", torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2")
# chat
text = "<image> <image> What's the difference between these two images? Please describe as much as you can."
response, history = chat_mllava(text, images, model, processor)
print("USER: ", text)
print("ASSISTANT: ", response)
# The image on the right has a larger number of wallets displayed compared to the image on the left. The wallets in the right image are arranged in a grid pattern, while the wallets in the left image are displayed in a more scattered manner. The wallets in the right image have various colors, including red, purple, and brown, while the wallets in the left image are primarily brown.
text = "How many items are there in image 1 and image 2 respectively?"
response, history = chat_mllava(text, images, model, processor, history=history)
print("USER: ", text)
print("ASSISTANT: ", response)
# There are two items in image 1 and four items in image 2.
Or, you can run the model without relying on the mantis codes, using pure hugging face transformers. See examples/run_mantis_hf.py for details.
Training
Training codes will be released soon.
Runs of TIGER-Lab Mantis-llava-7b on huggingface.co
7
Total runs
0
24-hour runs
0
3-day runs
3
7-day runs
-46
30-day runs
More Information About Mantis-llava-7b huggingface.co Model
Url of Mantis-llava-7b
Mantis-llava-7b huggingface.co Url
Provider of Mantis-llava-7b huggingface.co
Other API from TIGER-Lab
Updated:November 15 2024
Total runs:
12.3K
Run Growth:
594
Growth Rate:
4.84%
Total runs:
10.6K
Run Growth:
-3.6K
Growth Rate:
-34.41%
Total runs:
8.3K
Run Growth:
323
Growth Rate:
3.90%
Total runs:
3.2K
Run Growth:
2.7K
Growth Rate:
84.03%
Total runs:
2.6K
Run Growth:
2.5K
Growth Rate:
96.48%
Total runs:
1.4K
Run Growth:
120
Growth Rate:
8.64%
Total runs:
923
Run Growth:
-10.3K
Growth Rate:
-1116.36%
Total runs:
694
Run Growth:
0
Growth Rate:
0.00%
Total runs:
579
Run Growth:
-2.3K
Growth Rate:
-389.98%
Total runs:
498
Run Growth:
110
Growth Rate:
22.09%
Total runs:
317
Run Growth:
-149
Growth Rate:
-47.00%
Total runs:
268
Run Growth:
195
Growth Rate:
72.76%
Total runs:
262
Run Growth:
155
Growth Rate:
59.16%
Total runs:
228
Run Growth:
99
Growth Rate:
43.42%
Total runs:
165
Run Growth:
128
Growth Rate:
77.58%
Total runs:
156
Run Growth:
19
Growth Rate:
12.18%
Total runs:
156
Run Growth:
134
Growth Rate:
85.90%
Total runs:
153
Run Growth:
110
Growth Rate:
71.90%
Total runs:
151
Run Growth:
129
Growth Rate:
85.43%
Total runs:
139
Run Growth:
18
Growth Rate:
12.95%
Total runs:
129
Run Growth:
57
Growth Rate:
44.88%
Total runs:
91
Run Growth:
29
Growth Rate:
31.87%
Total runs:
57
Run Growth:
-4
Growth Rate:
-7.02%
Total runs:
54
Run Growth:
50
Growth Rate:
92.59%
Total runs:
47
Run Growth:
-18
Growth Rate:
-40.00%
Total runs:
46
Run Growth:
-327
Growth Rate:
-710.87%
Total runs:
26
Run Growth:
19
Growth Rate:
73.08%
Total runs:
24
Run Growth:
16
Growth Rate:
66.67%
Total runs:
23
Run Growth:
-13
Growth Rate:
-56.52%
Total runs:
21
Run Growth:
-69
Growth Rate:
-328.57%
Total runs:
20
Run Growth:
4
Growth Rate:
20.00%
Total runs:
19
Run Growth:
15
Growth Rate:
78.95%
Total runs:
19
Run Growth:
9
Growth Rate:
50.00%
Total runs:
18
Run Growth:
7
Growth Rate:
38.89%
Total runs:
17
Run Growth:
0
Growth Rate:
0.00%
Total runs:
16
Run Growth:
-7
Growth Rate:
-41.18%
Total runs:
16
Run Growth:
12
Growth Rate:
75.00%
Total runs:
14
Run Growth:
6
Growth Rate:
42.86%
Total runs:
13
Run Growth:
6
Growth Rate:
46.15%
Total runs:
13
Run Growth:
0
Growth Rate:
0.00%
Total runs:
13
Run Growth:
3
Growth Rate:
23.08%
Total runs:
12
Run Growth:
10
Growth Rate:
83.33%
Total runs:
11
Run Growth:
-6
Growth Rate:
-54.55%
Total runs:
11
Run Growth:
4
Growth Rate:
36.36%
Total runs:
10
Run Growth:
0
Growth Rate:
0.00%
Total runs:
9
Run Growth:
-12
Growth Rate:
-133.33%
Total runs:
7
Run Growth:
0
Growth Rate:
0.00%
Total runs:
6
Run Growth:
-5
Growth Rate:
-83.33%
Total runs:
6
Run Growth:
1
Growth Rate:
16.67%
Total runs:
5
Run Growth:
-40
Growth Rate:
-800.00%
Total runs:
5
Run Growth:
-23
Growth Rate:
-460.00%
Total runs:
4
Run Growth:
-1
Growth Rate:
-25.00%
Total runs:
3
Run Growth:
2
Growth Rate:
66.67%