Readme
Unofficial BLIP-2 demo and API
Note that this is an unofficial implementation of BLIP-2 that is not associated with Salesforce.
Usage
Blip-2 is a model that answers questions about images. To use it, provide an image, and then ask a question about that image. For example, you can provide the following image:
and then pose the following question:
What is this a picture of?
and get the output:
marina bay sands, singapore
.
Blip-2 is also capable of captioning images. This works by sending the model a blank prompt, though we have an explicit toggle for image captioning in the UI & API.
You can also provide Blip-2 with more context when asking a question. For example, given the following image:
you can provide the output of a previous Q&A as context in
question: ... answer: ...
format like so:
question: what animal is this? answer: panda
and then pose an additional question:
what country is this animal from?
and get the output:
china
Model description
BLIP-2 is a generic and efficient pre-training strategy that easily harvests development of pretrained vision models and large language models (LLMs) for vision-language pretraining. BLIP-2 beats Flamingo on zero-shot VQAv2 (65.0 vs 56.3), establishing new state-of-the-art on zero-shot captioning (on NoCaps 121.6 CIDEr score vs previous best 113.2). Equipped with powerful LLMs (e.g. OPT, FlanT5), BLIP-2 also unlocks the new zero-shot instructed vision-to-language generation capabilities for various interesting applications! Learn more at the official repo
Citation
@misc{https://doi.org/10.48550/arxiv.2301.12597,
doi = {10.48550/ARXIV.2301.12597},
url = {https://arxiv.org/abs/2301.12597},
author = {Li, Junnan and Li, Dongxu and Savarese, Silvio and Hoi, Steven},
keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models},
publisher = {arXiv},
year = {2023},
copyright = {Creative Commons Attribution 4.0 International}
}
