Total runs:
100.8K
Run Growth:
61.8K
Growth Rate:
61.30%
Sponsored by
APIMart
TIGER-Lab / Mantis-8B-Fuyu
Model's Last Updated: May 04 2024
text-generation
Introduction of Mantis-8B-Fuyu
Model Details of Mantis-8B-Fuyu
🔥 Mantis
Paper | Website | Github | Models | Demo
Summary
- Mantis-Fuyu is a Fuyu based LMM with interleaved text and image as inputs , train on Mantis-Instruct under academic-level resources (i.e. 36 hours on 16xA100-40G).
- Mantis is trained to have multi-image skills including co-reference, reasoning, comparing, temporal understanding.
- Mantis reaches the state-of-the-art performance on five multi-image benchmarks (NLVR2, Q-Bench, BLINK, MVBench, Mantis-Eval), and also maintain a strong single-image performance on par with CogVLM and Emu2.
Multi-Image Performance
Single-Image Performance
How to use
Installation
# This only installs minimum packages (torch, transformers, accelerate) for inference, no redundant packages are installed.
pip install git+https://github.com/TIGER-AI-Lab/Mantis.git
Run example inference:
from mantis.models.mllava import chat_mllava
from PIL import Image
import torch
image1 = "image1.jpg"
image2 = "image2.jpg"
images = [Image.open(image1), Image.open(image2)]
# load processor and model
from mantis.models.mfuyu import MFuyuForCausalLM, MFuyuProcessor
processor = MFuyuProcessor.from_pretrained("TIGER-Lab/Mantis-8B-Fuyu")
attn_implementation = None # or "flash_attention_2"
model = MFuyuForCausalLM.from_pretrained("TIGER-Lab/Mantis-8B-Fuyu", device_map="cuda", torch_dtype=torch.bfloat16, attn_implementation=attn_implementation)
generation_kwargs = {
"max_new_tokens": 1024,
"num_beams": 1,
"do_sample": False,
"pad_token_id": processor.tokenizer.eos_token_id,
}
# chat
text = "Describe the difference of <image> and <image> as much as you can."
response, history = chat_mllava(text, images, model, processor, **generation_kwargs)
print("USER: ", text)
print("ASSISTANT: ", response)
text = "How many wallets are there in image 1 and image 2 respectively?"
response, history = chat_mllava(text, images, model, processor, history=history, **generation_kwargs)
print("USER: ", text)
print("ASSISTANT: ", response)
"""
USER: Describe the difference of <image> and <image> as much as you can.
ASSISTANT: The second image has more variety in terms of colors and designs. While the first image only shows two brown leather pouches, the second image features four different pouches in various colors and designs, including a purple one with a gold coin, a red one with a gold coin, a black one with a gold coin, and a brown one with a gold coin. This variety makes the second image more visually interesting and dynamic.
USER: How many wallets are there in image 1 and image 2 respectively?
ASSISTANT: There are two wallets in image 1, and four wallets in image 2.
"""
Training
See mantis/train for details
Evaluation
See mantis/benchmark for details
Citation
@inproceedings{Jiang2024MANTISIM,
title={MANTIS: Interleaved Multi-Image Instruction Tuning},
author={Dongfu Jiang and Xuan He and Huaye Zeng and Cong Wei and Max W.F. Ku and Qian Liu and Wenhu Chen},
publisher={arXiv2405.01483}
year={2024},
}
Runs of TIGER-Lab Mantis-8B-Fuyu on huggingface.co
39
Total runs
0
24-hour runs
-6
3-day runs
-4
7-day runs
26
30-day runs
More Information About Mantis-8B-Fuyu huggingface.co Model
Url of Mantis-8B-Fuyu
Mantis-8B-Fuyu huggingface.co Url
Provider of Mantis-8B-Fuyu huggingface.co
Other API from TIGER-Lab
Updated:November 15 2024
Total runs:
12.3K
Run Growth:
594
Growth Rate:
4.84%
Total runs:
10.6K
Run Growth:
-3.6K
Growth Rate:
-34.41%
Total runs:
8.3K
Run Growth:
323
Growth Rate:
3.90%
Total runs:
3.2K
Run Growth:
2.7K
Growth Rate:
84.03%
Total runs:
1.4K
Run Growth:
120
Growth Rate:
8.64%
Total runs:
1.3K
Run Growth:
1.2K
Growth Rate:
92.90%
Total runs:
923
Run Growth:
-10.3K
Growth Rate:
-1116.36%
Total runs:
694
Run Growth:
0
Growth Rate:
0.00%
Total runs:
579
Run Growth:
-2.3K
Growth Rate:
-389.98%
Total runs:
498
Run Growth:
110
Growth Rate:
22.09%
Total runs:
390
Run Growth:
268
Growth Rate:
68.72%
Total runs:
262
Run Growth:
155
Growth Rate:
59.16%
Total runs:
228
Run Growth:
99
Growth Rate:
43.42%
Total runs:
165
Run Growth:
128
Growth Rate:
77.58%
Total runs:
156
Run Growth:
19
Growth Rate:
12.18%
Total runs:
156
Run Growth:
134
Growth Rate:
85.90%
Total runs:
155
Run Growth:
99
Growth Rate:
60.74%
Total runs:
153
Run Growth:
110
Growth Rate:
71.90%
Total runs:
151
Run Growth:
129
Growth Rate:
85.43%
Total runs:
139
Run Growth:
18
Growth Rate:
12.95%
Total runs:
114
Run Growth:
103
Growth Rate:
90.35%
Total runs:
91
Run Growth:
29
Growth Rate:
31.87%
Total runs:
83
Run Growth:
-23
Growth Rate:
-27.71%
Total runs:
54
Run Growth:
50
Growth Rate:
92.59%
Total runs:
53
Run Growth:
36
Growth Rate:
67.92%
Total runs:
46
Run Growth:
-327
Growth Rate:
-710.87%
Total runs:
37
Run Growth:
-33
Growth Rate:
-89.19%
Total runs:
32
Run Growth:
-426
Growth Rate:
-1331.25%
Total runs:
29
Run Growth:
15
Growth Rate:
51.72%
Total runs:
27
Run Growth:
4
Growth Rate:
14.81%
Total runs:
26
Run Growth:
4
Growth Rate:
15.38%
Total runs:
21
Run Growth:
-69
Growth Rate:
-328.57%
Total runs:
19
Run Growth:
-3
Growth Rate:
-15.79%
Total runs:
17
Run Growth:
0
Growth Rate:
0.00%
Total runs:
17
Run Growth:
-4
Growth Rate:
-23.53%
Total runs:
14
Run Growth:
5
Growth Rate:
35.71%
Total runs:
14
Run Growth:
0
Growth Rate:
0.00%
Total runs:
14
Run Growth:
6
Growth Rate:
42.86%
Total runs:
13
Run Growth:
0
Growth Rate:
0.00%
Total runs:
13
Run Growth:
3
Growth Rate:
23.08%
Total runs:
11
Run Growth:
1
Growth Rate:
9.09%
Total runs:
11
Run Growth:
-6
Growth Rate:
-54.55%
Total runs:
10
Run Growth:
0
Growth Rate:
0.00%
Total runs:
9
Run Growth:
-12
Growth Rate:
-133.33%
Total runs:
8
Run Growth:
-4
Growth Rate:
-50.00%
Total runs:
7
Run Growth:
0
Growth Rate:
0.00%
Total runs:
7
Run Growth:
-10
Growth Rate:
-142.86%
Total runs:
6
Run Growth:
-9
Growth Rate:
-150.00%
Total runs:
6
Run Growth:
-1
Growth Rate:
-16.67%
Total runs:
6
Run Growth:
1
Growth Rate:
16.67%
Total runs:
5
Run Growth:
-40
Growth Rate:
-800.00%
Total runs:
5
Run Growth:
-2
Growth Rate:
-40.00%
Total runs:
5
Run Growth:
-23
Growth Rate:
-460.00%
Total runs:
4
Run Growth:
-1
Growth Rate:
-25.00%
Total runs:
3
Run Growth:
2
Growth Rate:
66.67%