xlangai / OpenCUA-32B

🌐 Website

📝 Paper

💻 Code

Introduction

Key Features

• Superior Computer-Use Capablity : Able to execute multi-step computer-use actions with effective planning and reasoning
• Multi-OS Support : Trained on demonstrations across Ubuntu, Windows, and macOS
• Visual Grounding : Strong GUI element recognition and spatial reasoning capabilities
• Multi-Image Context : Processes up to 3 screenshot history for better context understanding
• Reflective Reasoning : Enhanced with reflective long Chain-of-Thought that identifies errors and provides corrective reasoning

Performance

Online Agent Evaluation

OpenCUA models achieves strong performance on OSWorld-Verified . OPENCUA-32B achieves the best performance among all open-source models with an average success rate of 34.8%, outperforming prior baselines by large margins. It also closes the gap to proprietary Claude models.

OpenCUA scores are the mean of 3 independent runs.

GUI Grounding Performance

AgentNetBench (Offline Evaluation)

🚀 Quick Start

Installation & Download

First, install the required transformers dependencies:

conda create -n opencua python=3.10
conda activate opencua
pip install -r requirement.txt

Download the model weight from huggingface:

from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="xlangai/OpenCUA-32B",
    local_dir="OpenCUA-32B",                
    local_dir_use_symlinks=False  
)

🎯 GUI Grounding

The following code demonstrates how to use OpenCUA models for GUI grounding tasks:

import base64
import torch
from transformers import AutoTokenizer, AutoModel, AutoImageProcessor
from PIL import Image
import json

def encode_image(image_path: str) -> str:
    """Encode image to base64 string for model input."""
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode()

def load_opencua_model(model_path: str):
    """Load OpenCUA model, tokenizer, and image processor."""
    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
    model = AutoModel.from_pretrained(
        model_path, 
        torch_dtype="auto", 
        device_map="auto", 
        trust_remote_code=True
    )
    image_processor = AutoImageProcessor.from_pretrained(model_path, trust_remote_code=True)
    
    return model, tokenizer, image_processor

def create_grounding_messages(image_path: str, instruction: str):
    """Create chat messages for GUI grounding task."""
    system_prompt = (
        "You are a GUI agent. You are given a task and a screenshot of the screen. "
        "You need to perform a series of pyautogui actions to complete the task."
    )
    
    messages = [
        {"role": "system", "content": system_prompt},
        {
            "role": "user",
            "content": [
                {"type": "image", "image": f"data:image/png;base64,{encode_image(image_path)}"},
                {"type": "text", "text": instruction},
            ],
        },
    ]
    return messages

def run_inference(model, tokenizer, image_processor, messages, image_path):
    """Run inference on the model."""
    # Prepare text input
    input_ids = tokenizer.apply_chat_template(
        messages, tokenize=True, add_generation_prompt=True
    )
    input_ids = torch.tensor([input_ids]).to(model.device)
    
    # Prepare image input  
    image = Image.open(image_path).convert('RGB')
    image_info = image_processor.preprocess(images=[image])
    pixel_values = torch.tensor(image_info['pixel_values']).to(
        dtype=torch.bfloat16, device=model.device
    )
    grid_thws = torch.tensor(image_info['image_grid_thw'])
    
    # Generate response
    with torch.no_grad():
        generated_ids = model.generate(
            input_ids,
            pixel_values=pixel_values,
            grid_thws=grid_thws,
            max_new_tokens=512,
            temperature=0
        )
    
    # Decode output
    prompt_len = input_ids.shape[1]
    generated_ids = generated_ids[:, prompt_len:]
    output_text = tokenizer.batch_decode(
        generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
    )[0]
    
    return output_text

# Example usage
model_path = "xlangai/OpenCUA-32B"  # or other model variants
image_path = "screenshot.png"
instruction = "Click on the submit button"

# Load model
model, tokenizer, image_processor = load_opencua_model(model_path)

# Create messages and run inference
messages = create_grounding_messages(image_path, instruction)
result = run_inference(model, tokenizer, image_processor, messages, image_path)

print("Model output:", result)

🖥️ Computer Use Agent

OpenCUAAgent is developed in the OSWorld environment based on OpenCUA models. It iteratively perceives the environment via screenshots, produces reflective long CoT as inner monologue, and predicts the next action to be executed. OpenCUAAgent uses 3 images in total and L2 CoT format in default.

Command for running OpenCUA-7B and OpenCUA-32B in OSWorld:

    python run_multienv_opencua.py \
        --headless \
        --observation_type screenshot \
        --model OpenCUA-32B \
        --result_dir ./results --test_all_meta_path evaluation_examples/test_all_no_gdrive.json \
        --max_steps 100 \
        --num_envs 30  \
        --coordinate_type qwen25

Important Notes on Coordinate Systems

OpenCUA models use different coordinate systems depending on the base model:

• OpenCUA-Qwen2-7B : Outputs relative coordinates (0.0 to 1.0 range)

  # Example output: pyautogui.click(x=0.5, y=0.3)
  # x=0.5 means 50% from left edge, y=0.3 means 30% from top edge
  
  # Convert to absolute coordinates:
  def qwen2_relative_to_absolute(rel_x, rel_y, original_width, original_height):
      abs_x = int(rel_x * original_width)
      abs_y = int(rel_y * original_height)
      return abs_x, abs_y
  

• OpenCUA-7B and OpenCUA-32B (Qwen2.5-based): Output absolute coordinates after smart resize

  # Example output: pyautogui.click(x=960, y=324)  
  # These are coordinates on the smart-resized image, not the original image
  
  # Convert to original image coordinates:
  # Please refer to the smart_resize function in: https://github.com/huggingface/transformers/blob/67ddc82fbc7e52c6f42a395b4a6d278c55b77a39/src/transformers/models/qwen2_vl/image_processing_qwen2_vl.py#L55
  def qwen25_smart_resize_to_absolute(model_x, model_y, original_width, original_height):
      # First, calculate the smart-resized dimensions
      resized_height, resized_width = smart_resize(original_height, original_width, factor = 28, min_pixels = 3136, max_pixels = 12845056)
      
      # Convert model output to relative coordinates on original image
      rel_x = model_x / resized_width
      rel_y = model_y / resized_height
      
      # Then convert to absolute coordinates on original image
      abs_x = int(rel_x * original_width)
      abs_y = int(rel_y * original_height)
      return abs_x, abs_y
  

TODO

vLLM Support

We are actively working with the vLLM team to add support for OpenCUA models.

Workaround: For now, please use the standard transformers library as shown in the examples above. We will update this section once vLLM support becomes available.

Training Code

OpenCUA models are developed based on the training infrastructure of Kimi Team. We are developting the training pipeline based on the open-source infrastructure as well.

License

This project is licensed under the MIT License - see the LICENSE file in the root folder for details.

Research Use and Disclaimer

OpenCUA models are intended for research and educational purposes only .

Prohibited Uses

• The model may not be used for any purpose or activity that violates applicable laws or regulations in any jurisdiction
• Use for illegal, unethical, or harmful activities is strictly prohibited

Disclaimer

• The authors, contributors, and copyright holders are not responsible for any illegal, unethical, or harmful use of the Software, nor for any direct or indirect damages resulting from such use
• Use of the "OpenCUA" name, logo, or trademarks does not imply any endorsement or affiliation unless separate written permission is obtained
• Users are solely responsible for ensuring their use complies with applicable laws and regulations

Citation

If you use OpenCUA models in your research, please cite our work:

@misc{wang2025opencuaopenfoundationscomputeruse,
      title={OpenCUA: Open Foundations for Computer-Use Agents}, 
      author={Xinyuan Wang and Bowen Wang and Dunjie Lu and Junlin Yang and Tianbao Xie and Junli Wang and Jiaqi Deng and Xiaole Guo and Yiheng Xu and Chen Henry Wu and Zhennan Shen and Zhuokai Li and Ryan Li and Xiaochuan Li and Junda Chen and Boyuan Zheng and Peihang Li and Fangyu Lei and Ruisheng Cao and Yeqiao Fu and Dongchan Shin and Martin Shin and Jiarui Hu and Yuyan Wang and Jixuan Chen and Yuxiao Ye and Danyang Zhang and Dikang Du and Hao Hu and Huarong Chen and Zaida Zhou and Yipu Wang and Heng Wang and Diyi Yang and Victor Zhong and Flood Sung and Y. Charles and Zhilin Yang and Tao Yu},
      year={2025},
      eprint={2508.09123},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2508.09123}, 
}