A powerful CLI tool that enables AI agents to observe and control desktop applications using native OS accessibility trees.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install aoto
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install aoto using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is agent-desktop?
agent-desktop is a specialized utility designed to give AI agents eyes and hands within a desktop environment. Instead of relying on fragile coordinate-based clicking or visual recognition, this tool interfaces directly with native accessibility trees to provide a structured, JSON-based view of the UI. This makes it an essential component for developers building sophisticated Openclaw Skills that require interaction with legacy software, system settings, or any application lacking a formal API.
By converting complex graphical interfaces into a ref-based system, agent-desktop allows agents to perform precise actions like clicking buttons, toggling checkboxes, and reading text fields with high reliability. It acts as a robust bridge between the high-level reasoning of an AI agent and the low-level APIs of the operating system, currently supporting macOS with plans for broader platform compatibility.
agent-desktop Use Cases
- Automating data entry and form submission across desktop-only enterprise applications.
- Scraping information from GUI-based tools that do not offer web interfaces or APIs.
- Orchestrating complex cross-app workflows, such as moving data from a spreadsheet into a CRM.
- Performing automated accessibility audits and UI testing for native desktop software.
- Monitoring system notifications and responding to alerts in real-time within Openclaw Skills.
How agent-desktop Works
- The AI agent triggers a snapshot command to retrieve the current accessibility tree of a target application.
- agent-desktop returns a structured JSON object where interactive elements are assigned ephemeral reference IDs (e.g., @e1, @e2).
- The agent analyzes the JSON to identify the correct element reference based on its role, name, or properties.
- The agent issues a specific interaction command, such as click or type, using the identified reference ID.
- A verification snapshot is often performed to confirm the UI state has changed as expected before the agent continues its task.
agent-desktop Setup
To integrate this tool into your Openclaw Skills workflow, install it globally using your preferred package manager:
npm install -g agent-desktop
# or
bun install -g --trust agent-desktop
Note: On macOS, you must grant Accessibility permissions to your terminal or IDE in System Settings > Privacy & Security > Accessibility for the tool to function correctly.
agent-desktop Data Schema & Taxonomy
All commands output a consistent JSON envelope to ensure compatibility with automated parsing logic. Below is the standard response structure:
| Field | Description |
|---|---|
version |
The schema version of the JSON output. |
ok |
A boolean indicating if the command executed successfully. |
command |
The name of the command that was run (e.g., snapshot, click). |
data |
The primary payload, such as the UI tree or element state. |
error |
Detailed error object including a code and recovery suggestion if the command fails. |
Interactive elements in snapshots are identified by a ref property, which is used for all subsequent interaction commands until the next snapshot is taken.
agent-desktop Advanced Features
- Smart element filtering using the -i flag to focus only on interactive components and reduce token overhead.
- Multi-surface support allowing agents to capture and interact with open menus, dialogs, and the Notification Center.
- Batch processing capabilities to execute multiple GUI interactions in a single high-performance invocation.
- Idempotent interaction commands like check and uncheck to ensure predictable UI states.
- Deep system integration including clipboard management, window resizing, and process lifecycle control.
SKILL.md
Loading
Related Openclaw Skills
OpenClaw WeCom Channel
An enterprise-grade communication bridge that allows OpenClaw AI agents to send and receive messages directly through Enterprise WeChat (WeCom).
Pane MCP
Connect your AI agents to real-time financial data, including bank balances, transactions, and investments via the Pane MCP server.
Grafana Dashboard Reader
A specialized skill enabling AI agents to extract data from Grafana dashboard panels without needing to understand underlying query languages.
Bring! Recipe Browser
A specialized CLI tool for discovering and filtering recipe inspirations from the Bring! shopping app platform.
Tavily Search Integration
A high-performance search and research engine integration providing structured, LLM-optimized web data for your AI agents.
Heartbeat Scanner
Identify whether a digital profile is an Agent, Human, Cron, or Hybrid by analyzing the rhythmic heartbeat of its posting patterns.
Featured*
