Directly control your desktop environment including mouse, keyboard, and window management through the CUA computer server API.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install desktop-automation
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install desktop-automation using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Desktop Control?
The desktop-control skill empowers Openclaw Skills with the ability to navigate and interact with a host operating system natively. By utilizing the CUA computer server running on a local or remote port, the AI agent can perform GUI-based actions that were previously restricted to manual human input. This skill is essential for bridging the gap between high-level agent reasoning and low-level desktop execution, making it a cornerstone for complex automation workflows.
Whether you are on macOS, Linux, or Windows, this skill provides a unified interface to capture screen data and simulate user interactions. It transforms your local environment into a programmable workspace where Openclaw Skills can execute tasks across different applications, manage windows, and handle file operations without needing a dedicated application API.
Desktop Control Use Cases
- Automating data entry across legacy desktop applications that lack formal APIs.
- Taking periodic screenshots for automated monitoring or visual documentation.
- Orchestrating multi-app workflows, such as extracting data from a browser and pasting it into a local spreadsheet.
- Remote desktop management and troubleshooting via natural language commands.
How Desktop Control Works
- A CUA computer server is initialized on the host machine, typically listening on port 8000.
- Openclaw Skills establishes a connection to the server REST API using the configured CUA_SERVER_URL environment variable.
- When a task is initiated, the agent sends a POST request containing specific commands such as click, type_text, or screenshot.
- The CUA server translates these requests into physical mouse or keyboard events on the host operating system.
- The server returns a JSON response, which may include base64 encoded screen data or execution status, to the agent to inform the next step.
Desktop Control Setup
Install the CUA SDK and start the server using the following commands:
# Install the SDK
pip install cua-computer-sdk
# Start the server on the default port
cua-server start --port 8000
To verify that the server is responding correctly, use curl:
curl http://localhost:8000/status
Desktop Control Data Schema & Taxonomy
The skill utilizes a structured JSON command protocol to interact with the OS. Key commands and their expected inputs are summarized below:
| Command | Parameters | Result |
|---|---|---|
screenshot |
N/A | Returns base64 string of the current screen capture. |
left_click |
x, y |
Moves the cursor and performs a click at coordinates. |
type_text |
text |
Inputs a string of characters at the current focus. |
hotkey |
keys (Array) |
Simulates key combinations like ['ctrl', 'alt', 't']. |
launch |
app |
Opens a specific application by its system name. |
open |
path |
Opens a file or URL with the default system handler. |
Desktop Control Advanced Features
- Background service integration using systemd for Linux, launchd for macOS, or Task Scheduler for Windows.
- Security-first configuration with support for authentication tokens and specific interface binding.
- Multi-display support through explicit environment variables like DISPLAY for Unix systems.
- Comprehensive window state management, including the ability to maximize, minimize, and retrieve window identifiers.
SKILL.md
Loading
Related Openclaw Skills
Agent Manager
A unified multi-agent management platform featuring a Gemini-style UI, image upload support, and isolated local chat history.
iMessage for macOS
A specialized skill for sending iMessages and SMS via the native macOS Messages app using automated AppleScript commands.
GitHub Deploy Skill
A PowerShell-driven automation script designed to handle Git initialization, commits, and pushes to GitHub with optional repository creation.
AI News Workflow
An automated intelligence gathering skill that aggregates, classifies, and summarizes AI industry news into professional reports.
File Batcher
A powerful utility for performing bulk file operations like renaming, converting, and organizing files through automated scripts.
Quick Note
A lightning-fast note-taking and snippet storage utility that uses simple CLI commands to manage persistent Markdown records.
Featured*
