keras / sam3_pcs

Model Overview

SAM 3

The Segment Anything Model 3 (SAM 3) is a high-performance foundation model for promptable object segmentation in images. Building upon the breakthroughs of previous SAM iterations, SAM 3 is designed for real-time performance, superior mask quality, and improved zero-shot generalization across diverse visual domains.

Model Summary

SAM 3 follows the "Segment Anything" philosophy by providing a universal interface for segmentation via prompts such as points, bounding boxes, or previous masks. It features a decoupled architecture that separates heavy image encoding from lightweight prompt processing, allowing the model to generate masks in near real-time once an image embedding has been computed.

SAM3 promptable concept segmentation (PCS) segments objects in images based on concept prompts, which could be short noun phrases (e.g., “yellow school bus”), image exemplars, or a combination of both. SAM3 PCS takes such prompts and returns segmentation masks and unique identities for all matching object instances.

There are two ways to prompt: 1. Text prompt: A short noun phrase describing the concept to segment. 2. Box prompt: A box tells the model which part/crop of the image to segment.

These prompts can be used individually or together, but at least one of the prompts must be present. To turn off a particular prompt, simply exclude it from the inputs to the model.

This modular design allows users to swap backbones of varying sizes (Tiny, Small, Base, Large) depending on the hardware constraints and accuracy requirements.

References

• [SAM 3 Quickstart Notebook](Coming Soon)
• SAM 3 API Documentation
• KerasHub Beginner Guide
• KerasHub Model Publishing Guide
• Segment Anything 3 Technical Report

Installation

Keras and KerasHub can be installed using the following commands:

pip install -U -q keras-hub
pip install -U -q keras

The following table summarizes the different configurations available for SAM 3 in Keras Hub:

Example Usage

image_size = 128
batch_size = 2
input_data = {
    "images": np.ones(
        (batch_size, image_size, image_size, 3), dtype="float32",
    ),
    "prompts": ["ear", "head"],
    "boxes": np.ones((batch_size, 1, 4), dtype="float32"),  # XYXY format.
    "box_labels": np.ones((batch_size, 1), dtype="float32"),
}
sam3_pcs = keras_hub.models.SAM3PromptableConceptImageSegmenter.from_preset(
    "sam3_pcs"
)
outputs = sam3_pcs.predict(input_data)
scores = outputs["scores"]  # [B, num_queries]
boxes = outputs["boxes"]  # [B, num_queries, 4]
masks = outputs["masks"]  # [B, num_queries, H, W]

Example Usage with Hugging Face URI

image_size = 128
batch_size = 2
input_data = {
    "images": np.ones(
        (batch_size, image_size, image_size, 3), dtype="float32",
    ),
    "prompts": ["ear", "head"],
    "boxes": np.ones((batch_size, 1, 4), dtype="float32"),  # XYXY format.
    "box_labels": np.ones((batch_size, 1), dtype="float32"),
}
hf://keras/sam3_pcs = keras_hub.models.SAM3PromptableConceptImageSegmenter.from_preset(
    "hf://keras/sam3_pcs"
)
outputs = hf://keras/sam3_pcs.predict(input_data)
scores = outputs["scores"]  # [B, num_queries]
boxes = outputs["boxes"]  # [B, num_queries, 4]
masks = outputs["masks"]  # [B, num_queries, H, W]