An advanced automation suite providing programmatic control over mouse, keyboard, screen capture, and window management for desktop environments.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install desktop-control-custom
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install desktop-control-custom using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Desktop Control Skill?
The Desktop Control skill is a powerful extension designed for Openclaw Skills that enables comprehensive robotic process automation (RPA) capabilities directly on your local machine. It allows AI agents and developers to interact with any desktop application—even those without APIs—by simulating human-like peripheral inputs and visual perception.
This skill bridges the gap between digital logic and graphical interfaces, offering pixel-perfect mouse precision, high-speed keyboard input, and robust window management. By integrating this into Openclaw Skills, you can automate repetitive manual tasks, perform cross-application data entry, and conduct automated UI testing with ease.
Desktop Control Skill Use Cases
- Automating data entry into legacy software or applications lacking public APIs.
- Performing automated UI testing and quality assurance across different desktop environments.
- Orchestrating complex workflows that require switching between multiple windows and applications.
- Creating automated screen recording or monitoring tools for visual verification.
- Developing accessibility tools or custom keyboard shortcuts for specialized software.
How Desktop Control Skill Works
- The user initializes the DesktopController, which hooks into the underlying OS-level input drivers.
- The skill calculates precise screen coordinates based on current resolution or template matching using OpenCV.
- Automation commands are dispatched to the mouse or keyboard buffer with configurable durations and human-like movement curves.
- Screen state is monitored via real-time screenshots or color detection to verify the success of actions.
- Built-in failsafes monitor the mouse position; moving the cursor to a screen corner immediately terminates all active operations to ensure user safety.
Desktop Control Skill Setup
To integrate this capability into your Openclaw Skills environment, install the necessary Python dependencies for peripheral and image control:
pip install pyautogui pillow opencv-python pygetwindow
Ensure your terminal or IDE has the required permissions to control your screen and input devices in your OS settings.
Desktop Control Skill Data Schema & Taxonomy
The Desktop Control skill manages data through real-time state tracking and file-based captures.
| Data Type | Description | Format/Taxonomy |
|---|---|---|
| Screen Captures | Visual snapshots of the desktop or specific regions. | PIL Image / PNG Files |
| Coordinates | Absolute or relative screen mapping. | (x, y) Integer Tuples |
| Window Metadata | Registry of active applications and their states. | Window Titles (Strings) |
| Color Data | Pixel-level color identification. | RGB (r, g, b) Tuples |
| Input Logs | Sequential history of automation events. | Chronological Text Logs |
Desktop Control Skill Advanced Features
- Human-like movement paths using Bezier curves to bypass basic bot detection.
- OCR and Image Recognition via OpenCV to find buttons or icons without fixed coordinates.
- Multi-monitor support for complex setups with secondary display spanning.
- Approval Mode which pauses automation until a human operator confirms the next action.
- Customizable Typing Speed (WPM) to simulate realistic user interaction patterns.
SKILL.md
Loading
Related Openclaw Skills
Slash GraphQL
A powerful integration for managing fully-managed GraphQL backends and automating data workflows without infrastructure overhead.
Market Alert
Market Alert is a high-performance financial monitoring skill for AI agents that provides real-time stock tracking, anomaly notifications, and detailed market analysis.
Einstein Research — Backtest Engine
A programmatic framework for executing quantitative trading strategies against historical data to generate professional-grade performance metrics.
Finishing a Development Branch
A structured workflow skill that automates test verification, branch merging, pull request creation, and worktree cleanup for completed development tasks.
Warehouse Optimization
A comprehensive advisor for e-commerce warehouse management and inventory health analysis across multiple fulfillment models.
Fengshui Advisor
An expert spatial analysis tool that combines traditional Eight Mansions and Flying Stars Fengshui systems to provide personalized home and office layout guidance.
Featured*
