A powerful desktop automation skill that combines traditional Python libraries with advanced Vision AI to control any GUI environment.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install desktop-gui
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install desktop-gui using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Desktop GUI Automation?
The desktop-gui skill is a comprehensive solution for full desktop automation within the Openclaw Skills ecosystem. It leverages Python libraries like pyautogui and opencv alongside system utilities such as xdotool and scrot to simulate human interactions. By providing a bridge between AI agents and the operating system, it enables programmatic control over mouse movements, keyboard inputs, and window management.
Beyond basic scripting, this skill introduces a cutting-edge Vision Mode. By integrating with the Qwen3.5-27b vision model, the skill can 'see' and understand screen layouts without relying on static coordinates or fragile OCR. This allows for intelligent automation that adapts to UI changes, making it ideal for legacy systems and complex cross-application workflows.
Desktop GUI Automation Use Cases
- Automating desktop software such as Office suites or professional creative tools.
- Implementing game automation and botting for testing or repetitive tasks.
- Interacting with legacy systems that lack modern APIs or web interfaces.
- Orchestrating complex cross-application workflows that require switching between different desktop windows.
- Automated attendance or clock-in systems using visual button recognition.
How Desktop GUI Automation Works
- The AI agent triggers a screenshot of the current desktop using the scrot utility.
- In Traditional Mode, the system uses OCR or template matching to find specific UI elements.
- In Vision Mode, the screenshot is sent to a Qwen3.5 vision model to analyze the layout and identify target coordinates based on natural language prompts.
- The skill generates specific interaction commands (e.g., click, type, drag).
- Execution is carried out via xdotool or pyautogui to perform the physical action on the OS.
- A follow-up screenshot is often taken to verify the success of the action.
Desktop GUI Automation Setup
To get started with this skill in Openclaw Skills, install the core Python libraries and system dependencies:
# Install core Python libraries
pip3 install pyautogui pygetwindow pymsgbox opencv-python-headless pillow pytesseract
# Install required system tools
sudo apt-get install -y xdotool scrot tesseract-ocr tesseract-ocr-chi-sim tesseract-ocr-chi-tra
Desktop GUI Automation Data Schema & Taxonomy
The skill manages automation data and visual state through the following structure:
| Data Type | Implementation | Description |
|---|---|---|
| Screenshots | /tmp/screen.png |
Temporary storage for visual analysis and verification. |
| Input Simulation | xdotool / pyautogui | Handles the translation of logic into hardware-level events. |
| Vision Pipeline | Base64 Encoding | Converts images into data strings for LLM processing. |
| Window Metadata | pygetwindow | Tracks window titles, coordinates, and focus states. |
Desktop GUI Automation Advanced Features
- Vision Mode Integration: Uses native vision capabilities of Qwen3.5-27b to understand UI context without OCR.
- Failsafe Security: Includes a hardware-level override (moving mouse to corner) to instantly abort automated scripts.
- Multi-Window Management: Ability to find, activate, minimize, or close specific windows by title or PID.
- Hybrid Execution: Combines the precision of xdotool for Linux environments with the cross-platform ease of pyautogui.
SKILL.md
Loading
Related Openclaw Skills
UFO Research Workbench
A professional-grade framework for triaging, grading, and reporting on declassified UAP, UFO, and FOIA document releases from multiple official sources.
Meta-Entre Architecture Generator
A high-level orchestration skill that generates end-to-end meta-architectures for complex entrepreneurial ventures, from business models to technical agent stacks.
Chinese Holiday Checker
An intelligent utility for querying Chinese statutory holidays, weekends, and make-up workday arrangements via automated API integration.
Yazhou Enterprise Service Skill
A comprehensive enterprise service tool designed for government personnel to manage business inquiries, policy recommendations, and safety supervision in the Yazhou District.
Embedded Code Migration Check
A professional tool for systematically checking compatibility and managing risks when migrating embedded code between hardware platforms.
Guardian Auto-Healer
A lightweight watchdog and self-healing system designed to provide 24/7 health monitoring and automatic recovery for Openclaw environments.
METADATA
Featured*
