A specialized automation skill for controlling local user interface actions and desktop behavior on a gateway machine.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install ui-controller
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install ui-controller using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is UI Controller?
The UI Controller is a robust extension designed for the Openclaw Skills ecosystem, specifically built to bridge the gap between AI agents and local machine interfaces. It enables direct interaction with the host operating system's UI, allowing for seamless personal automation and programmatic desktop control from within the OpenClaw gateway environment.
By leveraging this skill, developers can create sophisticated workflows that manipulate windows, trigger system-level actions, and automate repetitive desktop tasks. As part of the growing library of Openclaw Skills, it provides a secure yet flexible way to bridge cloud-based logic with local execution on Windows platforms.
UI Controller Use Cases
- Automating repetitive local desktop interactions and software navigation.
- Triggering system-level UI actions through remote messaging or AI agent logic.
- Creating personal productivity shortcuts that interact directly with the gateway machine's interface.
- Orchestrating complex multi-app workflows that require local GUI focus.
How UI Controller Works
- The user sends a command or message that matches a predefined pattern defined in the OpenClaw task file.
- The OpenClaw gateway receives the request and identifies the UI Controller as the handling agent.
- The skill translates the high-level intent into specific local UI hooks or desktop commands.
- Actions are executed locally on the machine where the gateway is actively running.
- The system returns the execution status to the user through the Openclaw Skills interface.
UI Controller Setup
To install and configure the UI Controller on your Windows gateway, follow these steps:
# Navigate to your OpenClaw skills directory
cd openclaw/skills
# Clone or add the UI Controller skill
openclaw install ui-controller
Ensure your task.yaml or relevant configuration file includes the patterns you wish to trigger. Because this skill interacts with the desktop, ensure the gateway process has sufficient administrative permissions to manipulate local windows.
UI Controller Data Schema & Taxonomy
The UI Controller manages interactions based on pattern matching and system-level calls. Below is the metadata structure used for task execution:
| Property | Type | Description |
|---|---|---|
pattern |
String | The regex or keyword match used to trigger the UI action. |
action_type |
Enum | The specific desktop behavior (e.g., click, focus, keystroke). |
platform |
String | Fixed to Windows for current implementation. |
security_context |
String | Defines the permission level required for the local execution. |
UI Controller Advanced Features
- Pattern-based task matching for flexible command recognition within the Openclaw Skills framework.
- Direct local execution bypasses network latency for immediate UI responsiveness.
- Integration-ready for complex automation sequences involving other local system tools.
- Support for custom task files to extend desktop behavior without modifying core skill logic.
SKILL.md
Loading
Related Openclaw Skills
memelink
A fast and flexible CLI tool for generating memes and image macros directly from the terminal using the Memegen.link API.
Claw Messenger
A specialized communication bridge allowing AI agents to send and receive iMessage, RCS, and SMS messages via a secure WebSocket connection.
CIO Intelligence
A strategic intelligence operation skill for AI startups to gather market signals, monitor competitors, and deliver actionable briefings.
Japan News MCP
A comprehensive AI agent skill for fetching and searching real-time Japanese financial, corporate, and economic news via public RSS feeds.
pawr.link Quick Profile Manager
A high-speed interface for deploying and managing AI agent profiles on the pawr.link platform via x402 automated payments.
Subscription Cancel Call Script
A prompt-based tool designed to build firm, honest cancellation call scripts with refusal phrases and confirmation logs for recurring services.
Featured*
