diffusers / Florence2-image-Annotator

Florence-2 Image Annotator

A custom Modular Diffusers block that uses Florence-2 for image annotation tasks like segmentation, object detection, and captioning.

How to use

import torch
from diffusers import ModularPipeline
from diffusers.utils import load_image

# Load the block
image_annotator = ModularPipeline.from_pretrained(
    "diffusers/Florence2-image-Annotator",
    trust_remote_code=True
)
image_annotator.load_components(torch_dtype=torch.bfloat16)
image_annotator.to("cuda")

# Load an image
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg")
image = image.resize((1024, 1024))

# Generate a segmentation mask
output = image_annotator(
    image=image,
    annotation_task="<REFERRING_EXPRESSION_SEGMENTATION>",
    annotation_prompt="the car",
    annotation_output_type="mask_image",
).images[0]
output.save("car-mask.png")

Compose with Inpainting Pipeline

from diffusers import ModularPipeline

# Load the annotator
image_annotator = ModularPipeline.from_pretrained(
    "diffusers/Florence2-image-Annotator",
    trust_remote_code=True
)

# Get an inpainting workflow and insert the annotator
# repo_id = .. # you can use SDXL/flux/qwen any pipeline support Inpaint
inpaint_blocks = ModularPipeline.from_pretrained(repo_id).blocks.get_workflow("inpainting")
inpaint_blocks.sub_blocks.insert("image_annotator", image_annotator.blocks, 0)

# Initialize the combined pipeline
pipe = inpaint_blocks.init_pipeline()
pipe.load_components(torch_dtype=torch.float16, device="cuda")

# Inpaint with automatic mask generation
output = pipe(
    prompt=prompt,
    image=image,
    annotation_task="<REFERRING_EXPRESSION_SEGMENTATION>",
    annotation_prompt="the car",
    annotation_output_type="mask_image",
    num_inference_steps=30,
).images[0]
output.save("inpainted-car.png")

Supported Tasks

Output Types

Inputs

Outputs

Components

This block uses the following models from florence-community/Florence-2-base-ft :

• image_annotator : Florence2ForConditionalGeneration
• image_annotator_processor : AutoProcessor

Learn More

• Building Custom Blocks Guide
• Modular Diffusers Overview
• Modular Diffusers Custom Blocks Collection