Reinforcement learning (RL) (e.g., GRPO) helps with grounding because of its inherent objective alignment—rewarding successful clicks—rather than encouraging long textual Chain-of-Thought (CoT) reasoning. Unlike approaches that rely heavily on verbose CoT reasoning, GRPO directly incentivizes actionable and grounded responses. Based on findings from our
blog
, we share state-of-the-art GUI grounding models trained using GRPO.
Grounding Performance
We follow the standard evaluation protocol and benchmark our model on three challenging datasets. Our method consistently achieves the best results among all open-source model families. Below are the comparative results:
Model
Size
Open Source
ScreenSpot-V2
ScreenSpotPro
OSWORLD-G
OSWORLD-G-Refined
OpenAI CUA
—
❌
87.9
23.4
—
—
Claude 3.7
—
❌
87.6
27.7
—
—
JEDI-7B
7B
✅
91.7
39.5
54.1
—
SE-GUI
7B
✅
90.3
47.0
—
—
UI-TARS
7B
✅
91.6
35.7
47.5
—
UI-TARS-1.5*
7B
✅
89.7*
42.0*
52.8*
64.2*
UGround-v1-7B
7B
✅
—
31.1
—
36.4
Qwen2.5-VL-32B-Instruct
32B
✅
91.9*
48.0
46.5
59.6*
UGround-v1-72B
72B
✅
—
34.5
—
—
Qwen2.5-VL-72B-Instruct
72B
✅
94.00*
53.3
—
62.2*
UI-TARS
72B
✅
90.3
38.1
—
—
OpenCUA
7B
✅
92.3
50.0
55.3
68.3*
OpenCUA
32B
✅
93.4
55.3
59.6
70.2*
GTA1-2507 (Ours)
7B
✅
92.4
(∆ +2.7)
50.1
(∆ +8.1)
55.1
(∆ +2.3)
67.7
(∆ +3.5)
GTA1 (Ours)
7B
✅
93.4
(∆ +0.1)
55.5
(∆ +5.5)
60.1
(∆ +4.8)
68.8
(∆ +0.5)
GTA1 (Ours)
32B
✅
95.2
(∆ +1.8)
63.6
(∆ +8.3)
65.2
(∆ +5.6)
72.2
(∆ +2.0)
Note:
Model size is indicated in billions (B) of parameters.
A dash (—) denotes results that are currently unavailable.
A superscript asterisk (﹡) denotes our evaluated result.
UI-TARS-1.5 7B, OpenCUA-7B, and OpenCUA-32B are applied as our baseline models.
∆ indicates the performance improvement (∆) of our model compared to its baseline.
Agent Performance
OSWorld and OSWorld-Verified Benchmarks
We evaluate our models on the OSWorld and OSWorld-Verified benchmarks following the standard evaluation protocol. The results demonstrate strong performance across both datasets.
Agent Model
Step
OSWorld
OSWorld-Verified
Proprietary Models
Claude 3.7 Sonnet
100
28.0
—
OpenAI CUA 4o
200
38.1
—
UI-TARS-1.5
100
42.5
41.8
OpenAI CUA o3
200
42.9
—
Open-Source Models
Aria-UI w/ GPT-4o
15
15.2
—
Aguvis-72B w/ GPT-4o
15
17.0
—
UI-TARS-72B-SFT
50
18.8
—
Agent S w/ Claude-3.5-Sonnet
15
20.5
—
Agent S w/ GPT-4o
15
20.6
—
UI-TARS-72B-DPO
15
22.7
—
UI-TARS-72B-DPO
50
24.6
—
UI-TARS-1.5-7B
100
26.9
27.4
Jedi-7B w/ o3
100
—
51.0
Jedi-7B w/ GPT-4o
100
27.0
—
Agent S2 w/ Claude-3.7-Sonnet
50
34.5
—
Agent S2 w/ Gemini-2.5-Pro
50
41.4
45.8
Agent S2.5 w/ o3
100
—
56.0
Agent S2.5 w/ GPT-5
100
—
58.4
CoAct-1 w/o3 & o4mini & OpenAI CUA 4o
150
—
60.8
GTA1-7B-2507 w/ o3
100
45.2
53.1
GTA1-7B-2507 w/ GPT-5
100
—
61.0
GTA1-32B w/ o3
100
—
55.4
GTA1-32B w/ GPT-5
100
—
63.4
Note:
A dash (—) indicates unavailable results.
WindowsAgentArena Benchmark
We also evaluate our models on the WindowsAgentArena benchmark, demonstrating strong performance in Windows-specific GUI automation tasks.
Agent Model
Step
Success Rate
Kimi-VL
15
10.4
WAA
—
19.5
Jedi w/ GPT-4o
100
33.7
GTA1-7B-2507 w/ o3
100
47.9
GTA1-7B-2507 w/ GPT-5
100
49.2
GTA1-32B w/ o3
100
51.2
GTA1-32B w/ GPT-5
100
50.6
Note:
A dash (—) indicates unavailable results.
Inference
Below is a code snippet demonstrating how to run inference using a trained model.
from transformers import AutoTokenizer, AutoImageProcessor
from transformers.models.qwen2_vl.image_processing_qwen2_vl_fast import smart_resize
from PIL import Image
from io import BytesIO
import base64
import re
from vllm import LLM, SamplingParams
instruction="click start"
image_path="example.png"
CLICK_REGEXES = [
# pyautogui.click(x=123, y=456)
re.compile(r"click\s*\(\s*x\s*=\s*(\d+)\s*,\s*y\s*=\s*(\d+)\s*\)", re.IGNORECASE),
# pyautogui.click(123, 456) or click(123,456)
re.compile(r"click\s*\(\s*(\d+)\s*,\s*(\d+)\s*\)", re.IGNORECASE),
]
defformat_message(image_path,instruction):
SYSTEM_PROMPT = (
"You are a GUI agent. You are given a task and a screenshot of the screen. ""You need to perform a series of pyautogui actions to complete the task."
)
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": [
{"type": "image", "image": image_path},
{"type": "text", "text": instruction},
]},
]
text = prompt_tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
text2, n = re.subn(
r"<\|media_begin\|>.*?<\|media_end\|>",
"<|vision_start|><|image_pad|><|vision_end|>",
text,
flags=re.S
)
if n == 0:
raise RuntimeError("Cannot find <|media_begin|>...<|media_end|> token.")
return text2
defparse_xy_from_text(text: str):
if"click"notin text.lower():
return [-1, -1]
for rx in CLICK_REGEXES:
m = rx.search(text)
if m:
try:
returnint(m.group(1)), int(m.group(2))
except Exception:
continuereturn [-1,-1]
defconvert_pil_image_to_base64(image):
buffered = BytesIO()
image.save(buffered, format="PNG")
return base64.b64encode(buffered.getvalue()).decode()
llm = LLM(
model="Salesforce/GTA1-7B",
tokenizer="Salesforce/GTA1-7B",
tokenizer_mode="slow",
trust_remote_code=True,
dtype="bfloat16",
limit_mm_per_prompt={"image": 1},
tensor_parallel_size=1,
)
prompt_tok = AutoTokenizer.from_pretrained("Salesforce/GTA1-7B", trust_remote_code=True)
sp = SamplingParams(max_tokens=512, temperature=0.0)
tokenizer = llm.get_tokenizer()
processor=AutoImageProcessor.from_pretrained("Salesforce/GTA1-7B", trust_remote_code=True)
image = Image.open(image_path).convert('RGB')
resized_height, resized_width = smart_resize(
image.height,
image.width,
factor=processor.patch_size * processor.merge_size,
min_pixels=processor.min_pixels,
max_pixels=processor.max_pixels,
)
resized_image = image.resize((resized_width, resized_height))
messages = format_message(image_path, instruction)
response = llm.generate(
[{"prompt": messages, "multi_modal_data": {"image": [resized_image]}}],
sampling_params=sp
)[0].outputs[0].text
coordinates = parse_xy_from_text(response)
print(coordinates[0]/resized_width*image.width, coordinates[1]/resized_height*image.height)
Model Serving
Below is an example script for serving the model.
import torch
import os
# -------------------------# System / Torch defaults# -------------------------
os.environ.setdefault("TOKENIZERS_PARALLELISM", "false") # avoid CPU oversubscription
os.environ.setdefault("VLLM_USE_V1", "1")
os.environ.setdefault("VLLM_ENGINE_IN_BACKGROUND_THREAD", "0")
import base64
import re
from typing importDict, List, Unionfrom PIL import Image
from io import BytesIO
import traceback
import argparse
import asyncio
import requests
import ray
from ray import serve
from fastapi import FastAPI
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams
import uuid
N_REPLICAS = 2try:
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = Trueexcept Exception:
pass# -------------------------# IO helpers# -------------------------defpil_to_base64(img: Image.Image, format: str = "PNG") -> str:
buffer = BytesIO()
img.save(buffer, format=format)
img_bytes = buffer.getvalue()
img_b64 = base64.b64encode(img_bytes).decode("utf-8")
return img_b64
defdata_uri_to_pil(data_uri: str) -> Image.Image:
header, b64_str = data_uri.split(",", 1)
img_data = base64.b64decode(b64_str)
buffer = BytesIO(img_data)
img = Image.open(buffer)
return img
defextract_images(messages: List[Dict]) -> List[Image.Image]:
images = []
for msg in messages:
if msg.get("role") == "user":
for content in msg.get("content", []):
if content.get("type") in ["image", "image_url"]:
if content["type"] == "image":
images.append(data_uri_to_pil(content["image"]).convert("RGB"))
else:
images.append(data_uri_to_pil(content["image_url"]["url"]).convert("RGB"))
return images
# -------------------------# Prompt builder# -------------------------defbuild_prompt_with_template(tokenizer: AutoTokenizer, messages: List[Dict]) -> str:
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
text2, n = re.subn(
r"<\|media_begin\|>.*?<\|media_end\|>",
"<|vision_start|><|image_pad|><|vision_end|>",
text,
flags=re.S,
)
if n == 0:
raise RuntimeError("Did not find <|media_begin|>...<|media_end|> block in template.")
return text2
# -------------------------# Deployment# -------------------------defbuild_app(model_path: str, num_replicas: int, port: int):
api = FastAPI(title="GTA1-7B Multi-GPU Service (High-throughput)")
@serve.deployment( num_replicas=num_replicas, ray_actor_options={"num_gpus": 1, "num_cpus": 4}, max_ongoing_requests=16,)classGTA1Model:
def__init__(self, model_path: str):
gpu_ids = ray.get_gpu_ids()
self.gpu_id = gpu_ids[0] if gpu_ids else0print(f"🔍 Ray assigned GPU IDs: {gpu_ids}")
# Initialize vLLM within this replica (Ray sets CUDA_VISIBLE_DEVICES)print(f"🔄 Initializing vLLM on GPU {self.gpu_id}[ray id] from {model_path}")
ifnot torch.cuda.is_available():
raise RuntimeError("CUDA is not available")
self.llm = LLM(
model=model_path,
tokenizer=model_path,
tokenizer_mode="slow",
trust_remote_code=True,
dtype="bfloat16",
limit_mm_per_prompt={"image": 1},
max_model_len=32768,
tensor_parallel_size=1,
)
self.vllm_tokenizer = self.llm.get_tokenizer()
self.hf_tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
self.model_path = model_path
self.dtype = "bfloat16"print(f"✅ vLLM initialized successfully (Ray GPU Id: {self.gpu_id})")
# ------------ batching core ------------ @serve.batch(max_batch_size=8, batch_wait_timeout_s=0.1) # increase if GPU allowsasyncdef_generate_batch(self, payload: Union[Dict, List[Dict]]):
"""Build prompts, enforce single image, and call vLLM.generate."""ifisinstance(payload, dict):
list_of_payloads = [payload]
else:
list_of_payloads = payload
request_id = uuid.uuid4().hex[:8]
# --- Build per-sample prompt/image ---
prompts: List[str] = []
images_per_req: List[Image.Image] = []
error_results = []
early_exit = Falsefor p in list_of_payloads:
try:
messages = p["messages"]
imgs = extract_images(messages)
iflen(imgs) != 1:
raise RuntimeError(f"Exactly one image is required, got {len(imgs)}")
prompt_text = build_prompt_with_template(self.hf_tokenizer, messages)
# Sanity check on tokens: 1 <|image_pad|>, no <|media_placeholder|>
tok = self.vllm_tokenizer
id_imgpad = tok.encode("<|image_pad|>", add_special_tokens=False)[0]
id_media = tok.encode("<|media_placeholder|>", add_special_tokens=False)[0]
ids = tok.encode(prompt_text, add_special_tokens=False)
ifsum(i == id_imgpad for i in ids) != 1orany(i == id_media for i in ids):
raise RuntimeError("Prompt media tokens invalid after conversion")
prompts.append(prompt_text)
images_per_req.append(imgs[0])
except Exception as e:
early_exit = True
trace = traceback.format_exc()
error_results.append(
{
"response": "",
"error": {
"message": str(e),
"trace": trace,
'type_of_payload': str(type(payload)),
'type_of_list_of_payloads': str(type(list_of_payloads)),
'type_of_p': str(type(p)),
'p_keys': str(p.keys()) ifisinstance(p, dict) elsestr(p),
},
"usage": {},
"gpu_id": self.gpu_id
}
)
if early_exit:
return error_results
# --- vLLM generation ---
args_base = list_of_payloads[0]
sp = SamplingParams(
max_tokens=args_base.get("max_new_tokens", 512),
temperature=args_base.get("temperature", 0.0),
top_p=args_base.get("top_p", 0.9),
)
requests_list = [
{"prompt": pr, "multi_modal_data": {"image": [im]}}
for pr, im inzip(prompts, images_per_req)
]
outs = self.llm.generate(requests_list, sampling_params=sp)
tok = self.vllm_tokenizer
results: List[Dict] = []
for pr, o inzip(prompts, outs):
text = o.outputs[0].text if o.outputs else""
gen_tokens = len(o.outputs[0].token_ids) if (o.outputs andhasattr(o.outputs[0], 'token_ids')) elseNone
prompt_tokens = len(tok.encode(pr, add_special_tokens=False))
usage = {
"prompt_tokens": prompt_tokens,
"generated_tokens": gen_tokens if gen_tokens isnotNoneelseNone,
"total_tokens": (prompt_tokens + gen_tokens) if gen_tokens isnotNoneelseNone,
}
results.append({
"response": text,
"error": "",
"usage": usage,
"gpu_id": self.gpu_id,
'bs_size_in_this_request': f"{request_id}:{len(list_of_payloads)}"
})
return results
# Exposed single-call entry that joins the batchasyncdefcall_llm(self, payload: Dict):
try:
res = await self._generate_batch(payload)
return res
except Exception as e:
trace = traceback.format_exc()
return {"response": "", "error": {"message": str(e), "trace": trace}, "usage": {}, "gpu_id": self.gpu_id}
defhealth(self):
return {
"status": "ok",
"gpu_id": self.gpu_id,
"dtype": self.dtype,
"model_path": self.model_path,
}
model = GTA1Model.bind(model_path)
@serve.deployment(max_ongoing_requests=96) @serve.ingress(api)classGTA1App:
def__init__(self, model_handle):
self.model_deployment = model_handle
@api.get("/health")asyncdefhealth_all(self):
# Calling the same Serve handle N times does not guarantee each call hits a different replica
attempts = max(8, N_REPLICAS * 4) # oversample
calls = [self.model_deployment.health.remote() for i inrange(attempts)]
replies = await asyncio.gather(*calls)
# dedupe by replica_id (or by tuple(gpu_id))
seen = {}
for r in replies:
seen[r.get("gpu_id", f"unknown-{len(seen)}")] = r
iflen(seen) >= N_REPLICAS:
breakreturn {"replicas": list(seen.values())}
@api.post("/call_llm")asyncdefcall_llm(self, req: Dict):
returnawait self.model_deployment.call_llm.remote(req)
return GTA1App.bind(model)
# -------------------------# Main# -------------------------if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--model_path", type=str, default="Salesforce/GTA1-7B")
parser.add_argument("--host", type=str, default="0.0.0.0")
parser.add_argument("--port", type=int, default=3005)
parser.add_argument("--num_replicas", type=int, default=2)
args = parser.parse_args()
N_REPLICAS = args.num_replicas
ray.init(ignore_reinit_error=True)
print(f"🚀 Starting GTA1-7B service on {args.host}:{args.port}")
serve.start(detached=True, http_options={"host": args.host, "port": args.port})
app = build_app(args.model_path, args.num_replicas, args.port)
serve.run(app, name="GTA1-7B", route_prefix="/")
# Quick health sampletry:
r = requests.get(f"http://0.0.0.0:{args.port}/health", timeout=5)
print(r.json())
except Exception as e:
print("Health probe failed:", e)
Here is the example usage,
import argparse
import base64
import concurrent.futures
import json
import os
import re
from typing importDict, List, Tuplefrom gui_agent.agent.gta1.format_message import encode_numpy_image_to_base64, encode_image_bytes, smart_resize
import requests
from PIL import Image, ImageDraw
defimage_file_to_data_uri(image_path: str) -> str:
ifnot os.path.exists(image_path):
raise FileNotFoundError(f"Image not found: {image_path}")
withopen(image_path, "rb") as f:
b64 = base64.b64encode(f.read()).decode("utf-8")
# default to png; serverside only requires a data URI header then commareturnf"data:image/png;base64,{b64}"defbuild_messages(image_path: str, instruction: str, system_prompt: str) -> List[Dict]:
return [
{"role": "system", "content": system_prompt},
{
"role": "user",
"content": [
{"type": "image", "image": image_file_to_data_uri(image_path)},
{"type": "text", "text": instruction},
],
},
]
defcall_health(base_url: str, timeout: float = 10.0) -> Dict:
r = requests.get(f"{base_url}/health", timeout=timeout)
r.raise_for_status()
return r.json()
defcall_single(
base_url: str, image_path: str, instruction: str, system_prompt: str, max_new_tokens: int = 512, temperature: float = 0.0, top_p: float = 0.9, timeout: float = 120.0,
) -> List[Dict]:
payload = {
"messages": build_messages(image_path, instruction, system_prompt),
"max_new_tokens": max_new_tokens,
"temperature": temperature,
"top_p": top_p,
}
r = requests.post(f"{base_url}/call_llm", json=payload, timeout=timeout)
r.raise_for_status()
resp = r.json()
ifisinstance(resp, dict):
return [resp]
return resp
defcall_many_concurrent(
base_url: str, image_path: str, instruction: str, system_prompt: str, num_requests: int, concurrency: int, max_new_tokens: int = 512, temperature: float = 0.0, top_p: float = 0.9, timeout: float = 120.0,
) -> List[List[Dict]]:
results: List[List[Dict]] = []
def_one(i: int) -> List[Dict]:
# Vary instruction slightly so you can trace requests
instr = f"{instruction} [req {i+1}/{num_requests}]"return call_single(
base_url,
image_path,
instr,
system_prompt,
max_new_tokens,
temperature,
top_p,
timeout,
)
with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as pool:
futures = [pool.submit(_one, i) for i inrange(num_requests)]
for fut in concurrent.futures.as_completed(futures):
results.append(fut.result())
return results
defpretty_print_response(batch_results: List[Dict]) -> None:
ifisinstance(batch_results, dict):
batch_results = [batch_results]
for idx, item inenumerate(batch_results):
if item.get("error"):
print(f"[#{idx}] ERROR: {json.dumps(item['error'], ensure_ascii=False)})")
else:
usage = item.get("usage", {})
print(f"[#{idx}] gpu={item.get('gpu_id')} tokens={usage} text=\n{item.get('response','').strip()}\n")
CLICK_KWARGS_REGEX = re.compile(r"pyautogui\.click\(\s*x\s*=\s*(\d+)\s*,\s*y\s*=\s*(\d+)\s*\)")
CLICK_POSARGS_REGEX = re.compile(r"pyautogui\.click\(\s*(\d+)\s*,\s*(\d+)\s*\)")
defextract_clicks_from_text(text: str) -> List[Tuple[int, int]]:
clicks: List[Tuple[int, int]] = []
for x, y in CLICK_KWARGS_REGEX.findall(text or""):
clicks.append((int(x), int(y)))
for x, y in CLICK_POSARGS_REGEX.findall(text or""):
clicks.append((int(x), int(y)))
return clicks
defextract_clicks_from_results(result_items: List[Dict]) -> List[Tuple[int, int]]:
clicks: List[Tuple[int, int]] = []
ifisinstance(result_items, dict):
result_items = [result_items]
for item in result_items:
if item.get("error"):
continue
clicks.extend(extract_clicks_from_text(item.get("response", "")))
return clicks
defcompute_resized_dims_for_server_mapping(image_path: str) -> Tuple[int, int, int, int]:
with Image.open(image_path) as im:
width, height = im.size
resized_H, resized_W = smart_resize(
height,
width,
factor=28,
min_pixels=1000,
max_pixels=1000000000000,
)
return width, height, int(resized_W), int(resized_H)
defmap_clicks_to_original(clicks_resized: List[Tuple[int, int]], original_w: int, original_h: int, resized_w: int, resized_h: int) -> List[Tuple[int, int]]:
if resized_w == 0or resized_h == 0:
return []
scale_x = original_w / float(resized_w)
scale_y = original_h / float(resized_h)
mapped: List[Tuple[int, int]] = []
for x, y in clicks_resized:
mapped_x = int(round(x * scale_x))
mapped_y = int(round(y * scale_y))
mapped.append((mapped_x, mapped_y))
return mapped
defdraw_circles_on_image(image_path: str, points: List[Tuple[int, int]], output_path: str, radius: int = 8, color: Tuple[int, int, int] = (255, 0, 0), width: int = 3) -> None:
ifnot points:
returnwith Image.open(image_path).convert("RGB") as img:
drawer = ImageDraw.Draw(img)
for (x, y) in points:
left = x - radius
top = y - radius
right = x + radius
bottom = y + radius
drawer.ellipse([(left, top), (right, bottom)], outline=color, fill=(0,255,0), width=width)
img.save(output_path)
print(f"Annotated image saved to: {output_path} (points drawn: {len(points)})")
SYSTEM_PROMPT = (
"You are a GUI agent. You are given a task and a screenshot of the screen. ""You need to perform a series of pyautogui actions to complete the task."
)
defmain():
parser = argparse.ArgumentParser(description="Examples: single and batched inference against GTA1-7B Ray Serve.")
parser.add_argument("--host", type=str, default="http://localhost", help="Ray Serve host, e.g. http://localhost or http://IP")
parser.add_argument("--port", type=int, default=3005, help="Ray Serve port")
parser.add_argument("--image", type=str, required=False, default="example.jpg", help="Path to input image")
parser.add_argument("--instruction", type=str, default="click the icon in the bottom row, third from the left", help="User instruction")
parser.add_argument("--system", type=str, default=SYSTEM_PROMPT)
parser.add_argument("--mode", type=str, choices=["single", "batch", "health"], default="batch")
parser.add_argument("--num_requests", type=int, default=8, help="Number of requests in batch mode")
parser.add_argument("--concurrency", type=int, default=8, help="Max concurrent HTTP calls in batch mode")
parser.add_argument("--max_new_tokens", type=int, default=512)
parser.add_argument("--temperature", type=float, default=0.0)
parser.add_argument("--top_p", type=float, default=0.9)
parser.add_argument("--timeout", type=float, default=180.0)
args = parser.parse_args()
base_url = f"{args.host}:{args.port}"if args.mode == "health":
info = call_health(base_url, timeout=10.0)
print(json.dumps(info, indent=2))
returnif args.mode == "single":
result_list = call_single(
base_url=base_url,
image_path=args.image,
instruction=args.instruction,
system_prompt=args.system,
max_new_tokens=args.max_new_tokens,
temperature=args.temperature,
top_p=args.top_p,
timeout=args.timeout,
)
print(result_list)
pretty_print_response(result_list)
clicks_resized = extract_clicks_from_results(result_list)
if clicks_resized:
orig_w, orig_h, resized_w, resized_h = compute_resized_dims_for_server_mapping(args.image)
mapped_clicks = map_clicks_to_original(clicks_resized, orig_w, orig_h, resized_w, resized_h)
out_path = f"ray_serve/annotated.png"
draw_circles_on_image(args.image, mapped_clicks, out_path)
returnif args.mode == "batch":
print(f"Submitting {args.num_requests} requests with concurrency={args.concurrency}...")
batch_outs = call_many_concurrent(
base_url=base_url,
image_path=args.image,
instruction=args.instruction,
system_prompt=args.system,
num_requests=args.num_requests,
concurrency=args.concurrency,
max_new_tokens=args.max_new_tokens,
temperature=args.temperature,
top_p=args.top_p,
timeout=args.timeout,
)
for i, one_result inenumerate(batch_outs):
print(f"===== Result for request {i+1} =====")
pretty_print_response(one_result)
all_clicks_resized: List[Tuple[int, int]] = []
for one_result in batch_outs:
all_clicks_resized.extend(extract_clicks_from_results(one_result))
if all_clicks_resized:
orig_w, orig_h, resized_w, resized_h = compute_resized_dims_for_server_mapping(args.image)
mapped_clicks = map_clicks_to_original(all_clicks_resized, orig_w, orig_h, resized_w, resized_h)
out_path = f"ray_serve/annotated.png"
draw_circles_on_image(args.image, mapped_clicks, out_path)
returnif __name__ == "__main__":
main()
Ethical Considerations
This model is released for research and educational purposes. While our model demonstrates strong performance on GUI benchmarks, users should carefully evaluate its suitability for their specific use cases.
Important Considerations:
Accuracy Limitations:
Like all AI systems, this model may produce incorrect outputs or fail to accurately identify GUI elements in certain scenarios.
Safety and Security:
Exercise caution when deploying GUI automation agents, especially in production environments where incorrect actions could affect system integrity or data security.
Human Oversight:
We recommend maintaining appropriate human supervision when using this model for automated GUI interactions.
Compliance:
Users are responsible for ensuring their use of this model complies with applicable laws, regulations, and organizational policies.
Recommended Best Practices:
Thoroughly test the model in controlled environments before production deployment
Implement safeguards and error handling mechanisms
Consider the potential impact of automated actions on user systems and data
Regularly monitor and validate model performance in your specific domain
For further guidance on use cases, refer to our AUP and AI AUP.
Citation
If you're using any GTA model or find it helpful in your research, please cite it as follows:
@article{yang2025gta1guitesttimescaling,
title={GTA1: GUI Test-time Scaling Agent}, author={Yan Yang and Dongxu Li and Yutong Dai and Yuhao Yang and Ziyang Luo and Zirui Zhao and Zhiyuan Hu and Junzhe Huang and Amrita Saha and Zeyuan Chen and Ran Xu and Liyuan Pan and Silvio Savarese and Caiming Xiong and Junnan Li}, year={2025}, eprint={2507.05791}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2507.05791}, }
Runs of Salesforce GTA1-7B on huggingface.co
147
Total runs
0
24-hour runs
2
3-day runs
-1
7-day runs
-807
30-day runs
More Information About GTA1-7B huggingface.co Model
GTA1-7B huggingface.co is an AI model on huggingface.co that provides GTA1-7B's model effect (), which can be used instantly with this Salesforce GTA1-7B model. huggingface.co supports a free trial of the GTA1-7B model, and also provides paid use of the GTA1-7B. Support call GTA1-7B model through api, including Node.js, Python, http.
GTA1-7B huggingface.co is an online trial and call api platform, which integrates GTA1-7B's modeling effects, including api services, and provides a free online trial of GTA1-7B, you can try GTA1-7B online for free by clicking the link below.
Salesforce GTA1-7B online free url in huggingface.co:
GTA1-7B is an open source model from GitHub that offers a free installation service, and any user can find GTA1-7B on GitHub to install. At the same time, huggingface.co provides the effect of GTA1-7B install, users can directly use GTA1-7B installed effect in huggingface.co for debugging and trial. It also supports api for free installation.