briaai / fibo-scene-analyzer

FIBO Scene Analyzer [dev]

FIBO Scene Analyzer [dev] was developed to cover a wide range of image understanding tasks using Bria's scene description format. A scene description holds a general description of the image (type, style, lighting, composition, color palette and more) and a tree of objects. Each object has a name, a description, its geometry (bounding boxes, polygon masks, body keypoints), its colors, its relationships to other objects and fields for its kind: people, animals, text and other objects.

The same model also does grounding (detection, masks, and body keypoints for a named target, or for all objects), OCR with line boxes, and text-to-description tasks: it expands a short prompt to a full scene description, and it refines or edits a scene description from an instruction.

We built FIBO Scene Analyzer from Qwen3.6-35B-A3B via multi-stage SFT on a custom synthetic dataset, then GRPO reinforcement learning with task-specific rewards, then a final short SFT stage. For synthetic data, we developed a 12-step workflow that combines computer vision tools, general and specialist models to extract all possible structured information from the images.

Supported tasks

AR is the width divided by the height, rounded to two decimals.

With a scene description ( scene_contexts in generate() ), expand_object describes the object so that it fits the scene, with relationships to the objects of the scene. The scene must not contain the object itself.

SCENE_JSON is compacted JSON with coordinates and colors as tokens. prompt_json() in scene_analyzer.py writes a SceneDescription in this form, also after post_process() .

For the grounding tasks, a named target that is not in the image gives an empty list. Without a target, detect and mask return every object; clothing, footwear and accessories are separate targets named "ITEM on PERSON".

Scene detail levels

expand , refine and edit accept general , short , long and full . expand_object accepts short , long , full and full_mask .

How to get started

Download the code files of this repository, then:

from PIL import Image
from scene_analyzer import SceneAnalyzerModel, SceneAnalyzerTask, DetailLevel

model = SceneAnalyzerModel("briaai/fibo-scene-analyzer")  # Transformers, device_map="auto"
image = Image.open("photo.jpg").convert("RGB")

[description] = model.generate([image], SceneAnalyzerTask.CAPTION, detail_level=DetailLevel.FULL, num_samples=3)
description.post_process()  # coordinate and color tokens to numbers
print(description.general.description)
for object in description.objects or []:
    print(object.kind, object.name, object.bounding_boxes)

[people] = model.generate([image], SceneAnalyzerTask.DETECT, text_inputs=["person"])
[text] = model.generate([image], SceneAnalyzerTask.OCR)

To render an answer as an HTML page with the geometry drawn on the image:

from caption_visualizer import CaptionVisualizer

CaptionVisualizer().save("description.html", image, description)

Harnesses

To make it easier to work with the model and get the best results, we provide harness classes based on the Pydantic scene description format. The three harnesses have the same SceneAnalyzerModel.generate() interface and the same task logic; only the engine differs.

What each harness does:

• It builds the prompt in the trained format. A prompt that the model was not trained on, such as a detail level that a task does not accept, raises ValueError .
• It samples num_samples answers and returns the best valid one. A scene description with the most objects wins; for grounding tasks, the most confident answer wins. An input without a valid answer is sampled again ( retries ), and None is returned if all attempts fail.
• With num_iterations , it runs revision rounds: the model sees its answer and adds the objects that it missed. This works for <caption> and the whole-image grounding tasks.

vLLM

from scene_analyzer_vllm import SceneAnalyzerModel

model = SceneAnalyzerModel("briaai/fibo-scene-analyzer", gpu_memory_utilization=0.9)
descriptions = model.generate(images, SceneAnalyzerTask.CAPTION, num_samples=3)

The model needs approximately 70 GB of GPU memory in bfloat16. On GPUs with less memory, pass tensor_parallel_size .

OpenAI-compatible server

vllm serve briaai/fibo-scene-analyzer --max-model-len 32768 --limit-mm-per-prompt '{"image": 5}' \
    --reasoning-parser deepseek_r1

from scene_analyzer_server import SceneAnalyzerModel

model = SceneAnalyzerModel("http://localhost:8000/v1")
descriptions = model.generate(images, SceneAnalyzerTask.CAPTION, num_samples=3, on_result=print)

The client sends standard chat-completion requests, and the chat template of the checkpoint renders the trained prompt. Two request fields outside the OpenAI schema are necessary for full function. vLLM and SGLang support both:

• skip_special_tokens: false . The JSON structure and the coordinate and color values are special tokens, and servers remove special tokens from the output by default.
• chat_template_kwargs: {"thinking_effort": ...} selects a reasoning trace. Without it, the model answers directly.

Without a harness

The prompt is one user turn: the images, then the task tags. The assistant turn opens with an empty think block, and the model answers with JSON:

from transformers import AutoModelForImageTextToText, AutoProcessor

processor = AutoProcessor.from_pretrained("briaai/fibo-scene-analyzer")
model = AutoModelForImageTextToText.from_pretrained("briaai/fibo-scene-analyzer", dtype="auto", device_map="auto")
prompt = (
    "<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|><caption><full><|im_end|>\n"
    "<|im_start|>assistant\n<think>\n\n</think>\n\n"
)
inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=16384, do_sample=True, temperature=0.6, top_p=0.97, top_k=0)
answer = processor.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=False)

Decode with skip_special_tokens=False , then remove the final <|im_end|> .

Thinking

thinking_effort ( minimal , low , medium or high ) makes the model write a reasoning trace before the answer. Direct answers are the default. Only caption , expand , refine and edit are trained with traces, with these efforts:

Below is example of gradual prompt expansion using Chain-of-Thought.

Sampling

The harnesses sample with temperature 0.6 and top-p 0.97. Three samples per input ( num_samples=3 ) give the best scene descriptions; more samples do not help.

Scene description schema

data_model.py defines the schema with Pydantic. Pass SceneDescription.model_json_schema() to a constrained decoder, or use model_validate_json() to parse an answer. All fields except the ones marked required are optional, and the model leaves out a field that does not apply.

Structure

SceneDescription                       the answer of `caption`, `expand`, `refine`, `edit`
├── general: GeneralDescription        the image as a whole
└── objects: list[SceneObject]         the objects in the image
    ├── Object.children                objects that are part of the object or on it
    ├── Person.clothing / footwear / accessories: list[Object]
    ├── Person.eyes / hair / facial_hair: Object
    ├── Person.children, Animal.children: list[Object | Text]
    └── Text.lines: list[TextLine]

SceneObjectList                        the answer of detect, mask, keypoints, ocr,
└── objects: list[SceneObject]         describe and expand_object

SceneObject is one of Object , Person , Animal and Text , selected by kind . Repeated objects are grouped: a row of chairs is one object with several boxes and num_objects , not one object per chair.

General description

Fields of every object

location , position and relative_size replace the geometry in the lower detail levels. data_model.py calculates them from bounding_boxes and relative_distance ( make_long() , make_short() ), so a full scene description converts to the lower detail levels without the model.

Fields by kind

object

person

animal

text

A TextLine has the fields of every object and text ( required ). Its kind is base_object . In full scene descriptions, each line has its own bounding_boxes .

Field presence by detail level

In short and long , the children of each object move to the top-level list. The wearables stay under their person in long , and short leaves them out.

Geometry and colors

ObjectMask.decode(width, height) rasterizes the polygons to a boolean array.

In the raw answer, coordinates and color channels are tokens: "<coord_12.3>" and "<color_200>" . post_process() converts them to numbers in place.

The keypoint names ("left" and "right" are the person's own sides):

• Body: nose, neck, and left/right eye, ear, shoulder, elbow, wrist, hip, knee and ankle.
• Face: chin, nose tip, upper lip center, lower lip center, and left/right iris, eyebrow inner, eyebrow outer and mouth corner.
• Hands: left/right thumb tip, index fingertip, middle fingertip, ring fingertip and pinky fingertip.
• Feet: left/right heel, big toe and small toe.

Examples

The images show answers of FIBO Scene Analyzer [dev]. The images for the general caption, OCR, detection, masks, keypoints, prompt expansion and object expansion use the harness defaults: direct answers, temperature 0.6, top-p 0.97 and three samples per input. Their input photos are from Wikimedia Commons and Flickr, and the photo credits are at the end of this section.

Full caption

<caption><full> gives the complete scene description: the general description and every object with its geometry, colors and fields. Each image shows the input, the description drawn on the image and a part of the JSON.

More examples (5)

General caption

<caption><general> gives only the general description: image type, style, medium, color palette, lighting, composition and a short description.

More examples (4)

OCR

<ocr> reads each line of text, gives a box for each line, and tags each text object with its language. The model was primarily trained for English, but it is also capable of understanding other languages.

More examples (6)

Named object detection

Each number is one <detect>NAME prompt. The last name in each list is not in the image, and its answer is an empty list.

More examples (4)

Masks

<mask> without a target gives a polygon mask for each object. Clothing, footwear and accessories are separate objects, named "ITEM on PERSON".

More examples (4)

Keypoints

<keypoints> without a target gives each person with a box and named keypoints on the body, feet, hands and face.

More examples (4)

Prompt expansion

<expand><full> writes the scene description of a new image from a short prompt and an aspect ratio. The images show its layout: the box of each object, the boxes of clothing and footwear, and the keypoints of each person.

More examples (4)

Object expansion

<expand_object><full> describes one new object for a scene. With the scene description of an <expand> answer as context, it places the object in the scene and gives its relationships to the objects of the scene. The left side of each image shows the prompt and its scene. The right side shows the object prompt and the scene with the new object (purple) and the scene objects that it relates to (teal).

More examples (5)

Citations

@article{kachlon2026bbq,
  title={BBQ-to-Image: Numeric Bounding Box and Qolor Control in Large-Scale Text-to-Image Models},
  author={Eliran Kachlon and Alexander Visheratin and Nimrod Sarid and Tal Hacham and Eyal Gutflaish and Saar Huberman and Hezi Zisman and David Ruppin and Ron Mokady},
  journal={arXiv preprint arXiv:2602.20672},
  year={2026}
}