A comprehensive desktop automation suite for Openclaw Skills enabling programmatic control over mouse, keyboard, and display operations.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install desktop-control
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install desktop-control using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Desktop Control Skill?
The Desktop Control skill represents the most advanced automation toolkit available for the Openclaw Skills ecosystem. It allows AI agents to interact with the desktop environment with human-like flexibility and machine-like precision. By bridging the gap between high-level logic and low-level system interactions, this skill empowers developers to automate any application, regardless of whether it provides a native API.
Built on a foundation of robust Python libraries, it handles everything from smooth, bezier-curve mouse movements to complex multi-key hotkey combinations. Whether you are automating a legacy enterprise application or building a sophisticated UI testing bot, this skill provides the necessary primitives to interact with the screen, manage windows, and process visual data in real-time.
Desktop Control Skill Use Cases
- Automating repetitive data entry and form-filling across non-web applications.
- Creating automated walkthroughs and UI tests for desktop software suites.
- Managing complex multi-window workflows and monitor configurations programmatically.
- Building visual monitors that trigger actions based on screen color changes or image presence.
- Executing batch file operations involving drag-and-drop interactions in file explorers.
How Desktop Control Skill Works
- The Openclaw Skills environment initializes the DesktopController, setting up safety boundaries and failsafe triggers.
- The skill identifies target elements through either absolute pixel coordinates or visual pattern matching using OpenCV.
- Commands are dispatched to the system input buffer to simulate mouse clicks, scrolls, or keystrokes.
- Screen state is captured via high-resolution screenshots to verify the success of the interaction.
- Window management functions ensure the target application is focused and correctly positioned throughout the lifecycle.
Desktop Control Skill Setup
To get started with this skill, install the necessary system dependencies via pip:
pip install pyautogui pillow opencv-python pygetwindow
Once installed, you can initialize the controller within your automation script:
from skills.desktop_control import DesktopController
dc = DesktopController(failsafe=True)
Desktop Control Skill Data Schema & Taxonomy
The Desktop Control skill organizes interaction data using the following structures:
| Data Type | Description | Format |
|---|---|---|
| Coordinates | X and Y pixel positions from the top-left corner | (int, int) |
| RGB Color | Red, Green, and Blue color values | (r, g, b) |
| Screen Region | Defined box for localized screen capture | (left, top, width, height) |
| Window Metadata | Title, position, and state of application windows | String/Object |
| Keyboard Mapping | Standardized names for special and modifier keys | String (e.g., 'enter', 'ctrl') |
Desktop Control Skill Advanced Features
- Human-like movement algorithms using bezier curves to avoid detection by anti-bot mechanisms.
- Integrated OpenCV support for high-confidence image recognition and element locating.
- Global Failsafe mechanism allowing instant abort by moving the mouse to any screen corner.
- Approval Mode for critical workflows, requiring manual confirmation before any simulated input is executed.
- Advanced clipboard integration for seamless data transfer between the agent and the OS.
SKILL.md
Loading
Related Openclaw Skills
NEAR Phishing Detector
A security-focused AI agent tool for identifying and reporting phishing threats within the NEAR Protocol ecosystem.
NEAR Content Creator
An automated content generation engine for the NEAR Protocol ecosystem that creates threads, market digests, and technical tutorials.
NEAR Agent Skills
A comprehensive toolkit for NEAR Protocol that provides AI agents with gas optimization strategies and deep on-chain analytics.
OpenSoul Agent Skill
OpenSoul provides AI agents with an immutable, blockchain-based audit log for persistent memory, self-reflection, and economic autonomy.
Agentic Paper Digest
A sophisticated AI-driven pipeline that fetches, filters, and summarizes the latest research papers from arXiv and Hugging Face.
Origram
A programmatic photo sharing platform for bots that uses the L402 protocol for instant, automated Bitcoin Lightning payments.
Featured*
