BAAI / RoboBrain

RoboBrain

[CVPR 2025] RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete.

Introduction

In recent years, the rapid development of multimodal large language models (MLLMs) has significantly advanced the research progress of artificial general intelligence (AGI). By utilizing vast multimodal data from the internet and combining it with self-supervised learning techniques, MLLMs have demonstrated exceptional capabilities in visual perception and understanding human language instructions. However, despite their impressive performance in general tasks, MLLMs still face substantial challenges in embodied scenarios, particularly in long-horizon manipulation tasks.

In robotics, long-horizon manipulation tasks are one of the core capabilities for robots executing complex tasks. These tasks typically involve multiple steps and long-term interactions, such as "preparing a cup of tea in the kitchen" or "completing item sorting in a warehouse." Such tasks require robots not only to understand abstract instructions but also to convert those instructions into concrete actions. Specifically, the successful execution of long-horizon manipulation tasks relies on three core capabilities:

• Planning : Robots need to decompose complex abstract instructions into executable subtasks. For example, "lifting the teapot and pouring water into the cup" must be broken down into steps such as "approaching the teapot and lifting it," "moving the teapot to align the spout over the cup," and "tilting the teapot to pour water."

• Affordance Perception : Robots must accurately identify the actionable regions of objects, such as the handle or spout of a teapot, to ensure the precision of their actions.

• Trajectory Prediction : Robots need to predict the complete path from the starting point to the target position based on the task instructions, such as the movement trajectory from the current position to the handle of the teapot.

However, existing MLLMs exhibit significant shortcomings in these areas. For instance, when faced with the task of "lifting the teapot and pouring water into the cup," MLLMs may struggle to accurately decompose the task steps, identify the graspable regions of the teapot, or even predict the complete trajectory from the starting point to the target position. These limitations primarily stem from the current lack of large-scale, fine-grained datasets specifically designed for MLLMs and long-horizon manipulation tasks in robotics.

To address this gap, we propose ShareRobot —a high-quality heterogeneous dataset specifically designed for robotic manipulation tasks. ShareRobot annotates multidimensional information, including task planning, object affordance regions, and end-effector trajectories, providing a solid foundation for enhancing robotic capabilities. Based on ShareRobot, we developed RoboBrain , a unified embodied multimodal brain model that aims to enhance robots' capabilities in long-horizon manipulation tasks. Through carefully designed data ratios, multi-stage training strategies, and inputs of long videos and high-resolution images, RoboBrain achieves a cognitive leap from abstract task instructions to concrete action expressions, demonstrating its potential for practical applications in robotics.

RoboBrain consists of three key robotic capabilities for long-horizon manipulation tasks: planning , affordance perception , and trajectory prediction . Based on the ShareRobot dataset we have constructed, RoboBrain has achieved state-of-the-art performance in multiple robotic benchmarks through a well-designed multi-stage training process, realizing a cognitive leap from abstract instruction understanding to concrete action expression.

Usage

import torch
from transformers import AutoProcessor, AutoModelForPreTraining

model_id = "BAAI/RoboBrain"

print("Loading Checkpoint ...")
model = AutoModelForPreTraining.from_pretrained(
    model_id, 
    torch_dtype=torch.float16, 
    low_cpu_mem_usage=True, 
).to("cuda:0")

processor = AutoProcessor.from_pretrained(model_id)

# Define a chat history and use `apply_chat_template` to get correctly formatted prompt
# Each value in "content" has to be a list of dicts with types ("text", "image") 
messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What is shown in this image?"},
            {"type": "image", "url": "http://images.cocodataset.org/val2017/000000039769.jpg"},
        ],
    },
]

print("Processing input...")
inputs = processor.apply_chat_template(
    messages, 
    add_generation_prompt=True, 
    tokenize=True, 
    return_dict=True, 
    return_tensors="pt"
)

inputs = {k: v.to("cuda:0") for k, v in inputs.items()}

print("Generating output...")
output = model.generate(**inputs, max_new_tokens=250)
print(processor.decode(output[0][2:], skip_special_tokens=True))

Citation

@article{ji2025robobrain,
  title={RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete},
  author={Ji, Yuheng and Tan, Huajie and Shi, Jiayu and Hao, Xiaoshuai and Zhang, Yuan and Zhang, Hengyuan and Wang, Pengwei and Zhao, Mengdi and Mu, Yao and An, Pengju and others},
  journal={arXiv preprint arXiv:2502.21257},
  year={2025}
}