Total runs:
932.1K
Run Growth:
-220.3K
Growth Rate:
-23.63%
Sponsored by
Meshy
Salesforce / blip-image-captioning-base
Model's Last Updated: February 03 2025
image-to-text
Introduction of blip-image-captioning-base
Model Details of blip-image-captioning-base
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Model card for image captioning pretrained on COCO dataset - base architecture (with ViT base backbone).
|
|---|
| Pull figure from BLIP official repo |
TL;DR
Authors from the paper write in the abstract:
Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to videolanguage tasks in a zero-shot manner. Code, models, and datasets are released.
Usage
You can use this model for conditional and un-conditional image captioning
Using the Pytorch model
Running the model on CPU
Click to expand
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
# conditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt")
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
# >>> a photography of a woman and her dog
# unconditional image captioning
inputs = processor(raw_image, return_tensors="pt")
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
>>> a woman sitting on the beach with her dog
Running the model on GPU
In full precision
Click to expand
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base").to("cuda")
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
# conditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt").to("cuda")
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
# >>> a photography of a woman and her dog
# unconditional image captioning
inputs = processor(raw_image, return_tensors="pt").to("cuda")
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
>>> a woman sitting on the beach with her dog
In half precision (
float16
)
Click to expand
import torch
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base", torch_dtype=torch.float16).to("cuda")
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
# conditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt").to("cuda", torch.float16)
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
# >>> a photography of a woman and her dog
# unconditional image captioning
inputs = processor(raw_image, return_tensors="pt").to("cuda", torch.float16)
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
>>> a woman sitting on the beach with her dog
BibTex and citation info
@misc{https://doi.org/10.48550/arxiv.2201.12086,
doi = {10.48550/ARXIV.2201.12086},
url = {https://arxiv.org/abs/2201.12086},
author = {Li, Junnan and Li, Dongxu and Xiong, Caiming and Hoi, Steven},
keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution 4.0 International}
}
Runs of Salesforce blip-image-captioning-base on huggingface.co
1.9M
Total runs
0
24-hour runs
-68.5K
3-day runs
-139.0K
7-day runs
152.4K
30-day runs
More Information About blip-image-captioning-base huggingface.co Model
Url of blip-image-captioning-base
blip-image-captioning-base huggingface.co Url
Provider of blip-image-captioning-base huggingface.co
Other API from Salesforce
Updated:January 21 2025
Total runs:
543.5K
Run Growth:
23.9K
Growth Rate:
4.22%
Total runs:
539.0K
Run Growth:
8.2K
Growth Rate:
1.52%
Total runs:
461.9K
Run Growth:
-167.0K
Growth Rate:
-34.30%
Total runs:
425.7K
Run Growth:
-75.7K
Growth Rate:
-16.89%
Total runs:
221.3K
Run Growth:
17.1K
Growth Rate:
7.71%
Total runs:
180.0K
Run Growth:
35.3K
Growth Rate:
19.62%
Total runs:
144.0K
Run Growth:
-99.5K
Growth Rate:
-69.08%
Total runs:
127.2K
Run Growth:
-15.9K
Growth Rate:
-12.47%
Total runs:
123.3K
Run Growth:
33.7K
Growth Rate:
26.79%
Total runs:
97.1K
Run Growth:
37.6K
Growth Rate:
38.70%
Total runs:
93.6K
Run Growth:
48.3K
Growth Rate:
51.57%
Total runs:
48.2K
Run Growth:
23.5K
Growth Rate:
48.83%
Total runs:
37.0K
Run Growth:
37.0K
Growth Rate:
99.94%
Total runs:
33.4K
Run Growth:
-62.2K
Growth Rate:
-164.32%
Total runs:
32.9K
Run Growth:
11.8K
Growth Rate:
35.88%
Total runs:
32.8K
Run Growth:
28.0K
Growth Rate:
85.48%
Total runs:
28.1K
Run Growth:
-21.4K
Growth Rate:
-76.08%
Total runs:
25.4K
Run Growth:
6.1K
Growth Rate:
23.47%
Total runs:
24.2K
Run Growth:
18.9K
Growth Rate:
78.27%
Total runs:
21.4K
Run Growth:
-6.2K
Growth Rate:
-27.71%
Total runs:
21.2K
Run Growth:
-387.1K
Growth Rate:
-1824.88%
Total runs:
18.5K
Run Growth:
483
Growth Rate:
2.61%
Total runs:
16.9K
Run Growth:
1.1K
Growth Rate:
6.30%
Total runs:
12.9K
Run Growth:
3.9K
Growth Rate:
30.32%
Total runs:
12.4K
Run Growth:
9.4K
Growth Rate:
75.93%
Total runs:
12.3K
Run Growth:
-25.3K
Growth Rate:
-206.19%
Total runs:
11.6K
Run Growth:
-4.8K
Growth Rate:
-39.41%
Total runs:
11.5K
Run Growth:
4.5K
Growth Rate:
39.32%
Total runs:
10.8K
Run Growth:
-6.5K
Growth Rate:
-60.56%
Total runs:
7.2K
Run Growth:
4.3K
Growth Rate:
60.37%
Total runs:
6.6K
Run Growth:
-1.5K
Growth Rate:
-23.16%
Total runs:
4.6K
Run Growth:
-998
Growth Rate:
-21.67%
Total runs:
3.6K
Run Growth:
-2.6K
Growth Rate:
-68.45%
Total runs:
3.1K
Run Growth:
2.1K
Growth Rate:
65.56%
Total runs:
3.0K
Run Growth:
2.0K
Growth Rate:
66.23%
Total runs:
2.3K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.5K
Run Growth:
1.3K
Growth Rate:
75.31%
Total runs:
1.2K
Run Growth:
-271
Growth Rate:
-22.45%
Total runs:
1.2K
Run Growth:
413
Growth Rate:
34.59%
Total runs:
1.2K
Run Growth:
-4.8K
Growth Rate:
-416.31%
Total runs:
1.1K
Run Growth:
-703
Growth Rate:
-63.22%
Total runs:
1.1K
Run Growth:
864
Growth Rate:
80.90%
Total runs:
1.0K
Run Growth:
-290
Growth Rate:
-28.32%
Total runs:
963
Run Growth:
-760
Growth Rate:
-78.92%
Total runs:
930
Run Growth:
245
Growth Rate:
26.34%
Total runs:
819
Run Growth:
-21
Growth Rate:
-2.44%
Total runs:
816
Run Growth:
-49
Growth Rate:
-6.00%
Total runs:
756
Run Growth:
-293
Growth Rate:
-37.81%
Total runs:
744
Run Growth:
-394
Growth Rate:
-52.96%
Total runs:
722
Run Growth:
550
Growth Rate:
75.14%
Total runs:
709
Run Growth:
448
Growth Rate:
63.19%
Total runs:
693
Run Growth:
-390
Growth Rate:
-53.79%
Total runs:
670
Run Growth:
106
Growth Rate:
15.82%
Total runs:
666
Run Growth:
219
Growth Rate:
32.74%
Total runs:
632
Run Growth:
162
Growth Rate:
25.63%
Total runs:
536
Run Growth:
277
Growth Rate:
48.26%
Total runs:
532
Run Growth:
92
Growth Rate:
17.29%
Total runs:
527
Run Growth:
-168
Growth Rate:
-31.88%
Total runs:
414
Run Growth:
146
Growth Rate:
35.27%
Total runs:
392
Run Growth:
-18
Growth Rate:
-4.59%
Total runs:
354
Run Growth:
107
Growth Rate:
30.23%
Total runs:
328
Run Growth:
-152
Growth Rate:
-46.34%
Total runs:
309
Run Growth:
168
Growth Rate:
54.37%
Total runs:
293
Run Growth:
-19
Growth Rate:
-6.48%
Total runs:
286
Run Growth:
75
Growth Rate:
26.22%
Total runs:
281
Run Growth:
-7
Growth Rate:
-2.49%
Total runs:
277
Run Growth:
-372
Growth Rate:
-134.30%
Total runs:
231
Run Growth:
-261
Growth Rate:
-112.99%
Total runs:
220
Run Growth:
-15
Growth Rate:
-6.82%
Total runs:
173
Run Growth:
-88
Growth Rate:
-50.87%
Total runs:
173
Run Growth:
-91
Growth Rate:
-52.60%
Total runs:
169
Run Growth:
-255
Growth Rate:
-150.89%
Total runs:
168
Run Growth:
19
Growth Rate:
11.31%
Total runs:
165
Run Growth:
67
Growth Rate:
40.61%
Total runs:
150
Run Growth:
-7
Growth Rate:
-4.67%
Total runs:
142
Run Growth:
47
Growth Rate:
33.10%
Total runs:
129
Run Growth:
-23
Growth Rate:
-17.83%
Total runs:
124
Run Growth:
-8
Growth Rate:
-6.45%
Total runs:
122
Run Growth:
24
Growth Rate:
19.67%
Total runs:
102
Run Growth:
-638
Growth Rate:
-625.49%
Total runs:
98
Run Growth:
-3
Growth Rate:
-3.03%