A comprehensive desktop automation skill that enables AI agents to control the mouse, keyboard, and screen across multiple operating systems.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install pipx-desktop-agent
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install pipx-desktop-agent using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Desktop Control?
The Desktop Control skill is a powerful interface for Openclaw Skills that leverages PyAutoGUI to provide AI agents with full agency over a desktop environment. It allows for seamless interaction with graphical user interfaces by simulating human actions such as moving the cursor, typing text, and executing complex hotkey combinations.
Beyond basic input, this skill provides advanced computer vision capabilities, including screenshot analysis and OCR (Optical Character Recognition). This allows agents to 'see' the screen, locate specific UI elements by image or text, and interact with applications on Windows, macOS, and Linux with high precision and reliability.
Desktop Control Use Cases
- Automating repetitive data entry tasks across legacy desktop applications.
- Performing end-to-end testing of graphical user interfaces by simulating user workflows.
- Orchestrating multi-app workflows, such as transferring data from a web browser to a local spreadsheet.
- Assisting users with complex desktop configurations or troubleshooting via remote automation.
- Capturing and analyzing screen content for automated reporting or monitoring.
How Desktop Control Works
- The AI agent initiates a command through the
uvx desktop-agentCLI to perform a specific action. - The skill identifies the target category: mouse, keyboard, screen, message, or app control.
- For visual tasks, the skill captures screen data or utilizes OCR to identify coordinates of buttons or text labels.
- Commands are executed via PyAutoGUI, simulating the physical input required to interact with the OS.
- The skill returns a structured JSON response to the agent, confirming success or providing detailed error codes for recovery.
Desktop Control Setup
To get started with this skill, ensure you have the necessary environment and run the following command to see available options:
# Show all available commands for desktop automation
uvx desktop-agent --help
# Install or run specific mouse commands
uvx desktop-agent mouse --help
Desktop Control Data Schema & Taxonomy
The skill utilizes a standardized JSON response schema for all operations, ensuring predictable programmatic interaction for Openclaw Skills.
| Field | Type | Description |
|---|---|---|
success |
Boolean | Indicates if the command executed without errors. |
command |
String | The specific category and command executed (e.g., mouse.move). |
timestamp |
ISO8601 | The exact time the operation occurred. |
duration_ms |
Number | Execution time in milliseconds. |
data |
Object | Command-specific output (coordinates, window lists, etc.). |
error |
Object | Detailed error code and message if success is false. |
Desktop Control Advanced Features
- Image-based element location with adjustable confidence levels for high-precision clicking.
- Integrated OCR support to locate and interact with specific text strings on the screen.
- Cross-platform application management to open, focus, and list windows on Windows, macOS, and Linux.
- Safety fail-safes that allow users to abort automated operations by moving the mouse to a screen corner.
- Native support for complex hotkeys and modifier combinations (e.g.,
ctrl,shift,esc).
SKILL.md
Loading
Related Openclaw Skills
Feishu Image Sender
A specialized automation skill for sending local image files from your workspace directly to Feishu channels.
AI Coding Standards
A robust system for AI agents to enforce code quality red lines, manage progressive context loading, and maintain persistent plan tracking.
Multi-Agent Parallel Build
A high-performance orchestration framework for spawning multiple coding agents simultaneously to build complex, multi-component systems in parallel waves.
Social Media Platform Builder
A comprehensive framework for building automated social media management systems with plugin-based publishing and AI-driven content pipelines.
GitHub AI Trends
A specialized tool for generating formatted leaderboards of trending GitHub AI projects based on real-time star counts and activity.
kk-daily-backup-report
A mobile-optimized reporting skill that provides concise, categorized daily backup status updates for servers and personal data.
Featured*
