A comprehensive automation skill for Openclaw Skills that enables AI agents to interact with desktop environments through screen recognition and peripheral control.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install doorstep-screen-control
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install doorstep-screen-control using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Screen Control?
The Screen Control skill bridges the gap between AI agents and local desktop environments by leveraging OpenClaw Node, pyautogui, and OCR technology. It allows your agent to perceive the screen, identify visual elements, and execute precise mouse and keyboard actions. This capability is essential for Openclaw Skills that require interaction with legacy software, desktop applications, or any interface that lacks a traditional API.
By integrating visual feedback loops, the skill ensures that agents can navigate complex graphical user interfaces (GUIs) just like a human operator. Whether you are managing professional software or automating repetitive system tasks, this skill provides the necessary vision and motor functions for your AI-driven workflows.
Screen Control Use Cases
- Automating login sequences and form filling for desktop-based enterprise software.
- Managing social media backends and content publishing platforms through visual navigation.
- Launching and navigating specific local applications via system search and keyboard shortcuts.
- Performing batch file organization, renaming, and management tasks within the OS file explorer.
How Screen Control Works
- The agent captures the current screen state using screenshot and size commands to establish a visual baseline for the operation.
- The AI analyzes the screenshot to identify target elements using image matching, OCR text recognition, or relative pixel coordinates.
- Based on the analysis, the skill executes the required actions such as clicking buttons, typing text, or performing complex keyboard hotkeys.
- A follow-up screenshot is captured to verify the result of the action, ensuring the workflow has transitioned to the expected next state.
Screen Control Setup
To utilize this skill within Openclaw Skills, ensure the OpenClaw Node is correctly paired. Install the required Python dependencies on the host machine:
pip install pyautogui pillow mss pytesseract
If you require text recognition capabilities, ensure that Tesseract-OCR is installed and accessible in your system's PATH environment variable.
Screen Control Data Schema & Taxonomy
| Component | Description | Data Format |
|---|---|---|
| Visual Input | Captured frames of the current desktop state | PNG Image File |
| Screen Geometry | Resolution and coordinate system mapping | Width x Height (Integers) |
| Action Commands | Low-level instructions for peripherals | Python/pyautogui commands |
| Vision Data | Results from OCR and template matching | Text Strings & Bounding Boxes |
Screen Control Advanced Features
- Intelligent image template matching using the locate command to find specific UI icons or buttons.
- Vision-based text interaction through findtext and clicktext OCR functions for dynamic layouts.
- Support for complex macro sequences including drag-and-drop operations and multi-key hotkey combinations.
- Built-in safety protocols including a physical fail-safe (moving mouse to corner) and user confirmation prompts for destructive actions.
SKILL.md
Loading
Related Openclaw Skills
Smooth Browser Use
A sophisticated interaction framework for AI agents to perform human-like browser navigation, reliable text entry, and proactive state recovery.
RollingGo Flight Search
A real-time flight search and price comparison engine for AI agents to find global airfare options.
Alibaba Cloud ICP Filing Data Query
A technical skill for automating the retrieval of ICP filing information and risk alerts from Alibaba Cloud.
Free Generation Image
A powerful AI image generation skill that transforms text prompts into stunning visual assets and 1080p MP4 videos using remote GPU nodes.
Agent Creator
A specialized skill for guided creation of new Openclaw agents and their integration into collaborative group environments.
Dom Perignon Strategy Analyst
A specialized technical guide for analyzing the business model, scarcity marketing, and luxury market positioning of the world's premier vintage champagne brand.
Featured*
