keras / blip2_opt_6.7b

Model Overview

Model Summary

BLIP-2 (Bootstrapping Language-Image Pre-training) is a generic and efficient pre-training strategy that bridges the modality gap between frozen image encoders and frozen large language models (LLMs). It was introduced by Salesforce in the paper "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models".

The architecture consists of three main components: a frozen image encoder (EVA-CLIP ViT-g/14), a lightweight querying transformer (Q-Former) that acts as an information bottleneck to extract the most relevant visual features, and a frozen LLM (like OPT or Flan-T5) that handles the text generation. Because the heavy vision and language models are kept frozen during pre-training, BLIP-2 achieves state-of-the-art performance on various vision-language tasks with significantly fewer trainable parameters than existing methods.

Key Features:

• State-of-the-art vision-language pre-training method
• Combines frozen EVA-CLIP vision encoder with frozen LLMs (OPT, Flan-T5)
• Highly efficient pre-training via the lightweight Q-Former
• Capable of visual question answering, image captioning, and conversational image understanding

Model Details

• Model Family: BLIP-2
• Architecture: Vision-Language Model (EVA-CLIP ViT + Q-Former + LLM)
• Developer: Salesforce
• Paper: BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Source: Salesforce BLIP-2 Collection
• Task Type: Image-to-Text / Vision-Language Generation
• Max Sequence Length: Varies by LLM (typically 512+ tokens)

Architecture Details (BLIP-2 General)

• Vision Encoder: EVA-CLIP ViT-g/14 (Frozen)
• Vision Encoder Parameters: ~1 Billion
• Q-Former: Querying Transformer (Trainable)
• Q-Former Parameters: ~188 Million
• Language Model: Varies (OPT-2.7B, OPT-6.7B, Flan-T5-XL, Flan-T5-XXL) (Frozen)
• Total Parameters: ~3B to ~12B (depending on the LLM variant)

• [BLIP2 Quickstart Notebook](coming soon..!)
• [BLIP2 API Documentation](coming soon..!)
• BLIP2 Model Card
• BLIP2 Technical Paper
• KerasHub Beginner Guide
• KerasHub Model Publishing Guide

Installation

Keras and KerasHub can be installed with:

pip install -U -q keras-hub
pip install -U -q keras

Jax, TensorFlow, and Torch come preinstalled in Kaggle Notebooks. For instructions on installing them in another environment see the Keras Getting Started page.

Preset Table

Example Usage

BLIP-2 can be used for various vision-language tasks such as image captioning and visual question answering (VQA). Depending on the preset you choose, you will use either BLIP2CausalLM (for OPT models) or BLIP2Seq2SeqLM (for Flan-T5 models).

The BLIP2Seq2SeqLM class supports BLIP-2 variants that use Flan-T5 as their base language model. This architecture requires passing your text prompt to the encoder using the "encoder_text" key.

import keras
import keras_hub
import numpy as np
from PIL import Image
import requests

model = keras_hub.models.BLIP2Seq2SeqLM.from_preset("blip2_opt_6.7b")

image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")
image_array = np.array(image)

vqa_input = {
    "images": image_array,
    "encoder_text": ["Question: what is in the picture? Answer:"]
}
print(model.generate(vqa_input))

caption_input = {
    "images": image_array,
    "encoder_text": ["A picture of"]
}
print(model.generate(caption_input))

Example Usage with Hugging Face URI

BLIP-2 can be used for various vision-language tasks such as image captioning and visual question answering (VQA). Depending on the preset you choose, you will use either BLIP2CausalLM (for OPT models) or BLIP2Seq2SeqLM (for Flan-T5 models).

The BLIP2Seq2SeqLM class supports BLIP-2 variants that use Flan-T5 as their base language model. This architecture requires passing your text prompt to the encoder using the "encoder_text" key.

import keras
import keras_hub
import numpy as np
from PIL import Image
import requests

model = keras_hub.models.BLIP2Seq2SeqLM.from_preset("hf://keras/blip2_opt_6.7b")

image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")
image_array = np.array(image)

vqa_input = {
    "images": image_array,
    "encoder_text": ["Question: what is in the picture? Answer:"]
}
print(model.generate(vqa_input))

caption_input = {
    "images": image_array,
    "encoder_text": ["A picture of"]
}
print(model.generate(caption_input))