A scriptable Linux X11 desktop control tool for automating GUI interactions via screenshots and simulated input.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install claw-mouse
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install claw-mouse using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is claw-mouse?
claw-mouse is a specialized utility designed for Linux X11 environments that enables programmatic control over the graphical user interface. By acting as a bridge between high-level automation scripts and low-level system events, it allows for the creation of sophisticated vision-loop agents. This skill is a vital addition to the ecosystem of Openclaw Skills for developers who need to automate desktop applications that lack native APIs.
Under the hood, the skill leverages the power of scrot for high-quality screen capture and xdotool for precise mouse and keyboard simulation. This combination allows an AI agent to see the current state of the desktop and perform actions like clicking buttons, typing text, or switching windows, effectively mimicking human interaction with the OS.
claw-mouse Use Cases
- Vision-based UI testing for native Linux desktop applications.
- Automating legacy software workflows that do not provide an API or CLI.
- Building autonomous agents that interact with graphical tools like image editors or IDEs.
- Capturing periodic screenshots of desktop activity for monitoring or data logging.
- Managing window states and application focus within an X11 session.
How claw-mouse Works
- The automation process begins by invoking a screenshot command to capture the current visual state of the X11 desktop.
- The captured image is processed by an AI model or script to identify UI elements and target coordinates.
- The skill uses the desktopctl.py script to send specific input events, such as mouse clicks or key presses, to the system.
- If necessary, the skill can query existing windows or activate specific applications to ensure the target UI is in focus.
- The sequence repeats in a loop, allowing the agent to react to visual changes on the screen in real-time.
claw-mouse Setup
To use this among your other Openclaw Skills, you must first install the required system dependencies on a Linux machine running X11:
sudo apt-get update
sudo apt-get install -y xdotool scrot python3
Once the binary requirements are met, you can execute commands directly via the provided script:
# Capture the screen
python3 desktopctl.py screenshot
# Click at specific coordinates and type text
python3 desktopctl.py click 800 600
python3 desktopctl.py type "Openclaw Skills automation"
claw-mouse Data Schema & Taxonomy
The skill manages interaction through a simple CLI interface and environment variables. The primary data outputs and inputs are organized as follows:
| Component | Type | Description |
|---|---|---|
desktopctl.py |
Script | The main CLI controller for mouse, keyboard, and screen events. |
screenshot.png |
File | The default output image generated when the screenshot command is used. |
DISPLAY |
Env Var | Defines which X display the skill should interact with (e.g., :0). |
XAUTHORITY |
Env Var | Path to the X authority file required for permissions in certain environments. |
claw-mouse Advanced Features
- Support for complex keyboard shortcuts and key combinations using xdotool syntax.
- Ability to run in headless or daemon environments by explicitly setting DISPLAY and XAUTHORITY flags.
- Window management capabilities including listing open windows and forcing activation/focus.
- Seamless integration into multi-step vision-action loops within the Openclaw Skills framework.
- Low-latency interaction with the X11 server for responsive automation.
SKILL.md
Loading
Related Openclaw Skills
Agent Synthesizer
A specialized performance enhancement skill designed to maximize AI agent autonomy and execution throughput through standardized GitHub-first workflows.
AADA (Agent-to-Agent Advertising)
A specialized tool for creating and delivering personality-driven promotional messages from one agent to the Moltbook audience.
MythosForge
Commission specialized AI agents for professional-grade deliverables paid via USDC micropayments on the Base network.
Plati MCP Search
A specialized search automation tool for finding the most affordable and trustworthy digital product subscriptions on the Plati marketplace.
Web Accessibility Auditor
A professional auditing tool for ensuring web frontends meet WCAG 2.1 accessibility standards.
API Documentation Generator
Automate the creation of professional, OpenAPI-compatible documentation for REST, GraphQL, and WebSocket APIs using Openclaw Skills.
Featured*
