A comprehensive automation suite providing pixel-perfect control over mouse, keyboard, and screen operations for AI agents.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install control
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install control using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Desktop Control Skill?
The Desktop Control Skill is the primary interface for enabling AI agents to interact directly with a desktop environment. As a core component within the ecosystem of Openclaw Skills, it provides the low-level hooks needed for mouse positioning, keyboard input simulation, and advanced window management. By integrating this skill, developers can extend their agent's capabilities beyond the terminal and into the visual workspace.
This library excels at bridging the gap between automated reasoning and GUI interaction. Whether it is navigating complex legacy software or performing high-speed form filling, this addition to Openclaw Skills ensures your agent has the precision and safety mechanisms required for professional-grade desktop automation.
Desktop Control Skill Use Cases
- Automating repetitive data entry across desktop forms and legacy applications.
- Creating visual documentation by capturing specific screen regions or windows automatically.
- Executing complex multi-key shortcuts and hotkey sequences for software development workflows.
- Building automated UI testing scripts that rely on image recognition to find elements.
- Managing file systems through drag-and-drop operations and multi-select keyboard modifiers.
How Desktop Control Skill Works
- The AI agent initializes the DesktopController, which loads the necessary drivers for OS-level input simulation.
- The skill maps the screen coordinate system, starting from the top-left (0,0) corner, to translate logical commands into physical cursor movements.
- Keyboard inputs are processed through a configurable buffer that supports both instant typing and human-like speed simulations.
- For visual tasks, the skill captures the current screen state and uses image processing to identify UI components dynamically.
- All operations are wrapped in safety layers, including an emergency failsafe that aborts tasks if the cursor is moved to a screen corner.
Desktop Control Skill Setup
To get started with this specific set of Openclaw Skills, you must first install the core automation dependencies:
pip install pyautogui pillow opencv-python pygetwindow
Once installed, initialize the controller within your Python environment to begin automating desktop tasks:
from skills.desktop_control import DesktopController
dc = DesktopController(failsafe=True)
Desktop Control Skill Data Schema & Taxonomy
The Desktop Control Skill organizes data using coordinate systems, image buffers, and window metadata. Below is the primary data structure utilized by the skill:
| Data Element | Type | Description |
|---|---|---|
| Coordinates | (int, int) | Absolute (x, y) pixel values on the primary or secondary display. |
| Screen Region | (l, t, w, h) | Tuple defining left, top, width, and height for targeted captures. |
| Pixel Color | (r, g, b) | RGB color values extracted from specific screen locations. |
| Window Object | String/Handle | Metadata including window titles and visibility states for management. |
Desktop Control Skill Advanced Features
- Smooth bezier-curve movement paths for natural, human-like mouse interactions.
- Integrated OpenCV support for template matching and image-based element detection.
- Approval mode to require manual user confirmation before high-stakes automation actions.
- Multi-monitor support to handle extended desktops and complex coordinate mappings.
- Configurable WPM (Words Per Minute) settings to bypass or mimic standard typing speeds.
SKILL.md
Loading
Related Openclaw Skills
Spotify History
A comprehensive integration for accessing Spotify listening history, top artists, and personalized music recommendations through the Spotify Web API.
Answer Overflow
Access indexed Discord support channels to find community-driven solutions to technical problems.
Session Wrap Up
An end-of-session protocol that preserves context, updates long-term memory, and synchronizes workspace changes to ensure continuity.
Giga Coding Agent
A high-performance skill for running background-first AI coding agents like Codex and Claude Code with directory isolation and parallel processing.
Find Skills
A discovery and installation engine for extending AI agent capabilities through the open skills ecosystem.
Session Memory
Bridges the gap between transient chat logs and long-term memory by summarizing sessions and enabling keyword-based history search.
Featured*
