A powerful macOS automation skill that uses Apple Vision OCR to visually interact with any graphical desktop application.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install mac-use
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install mac-use using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Mac Use?
The Mac Use skill enables AI agents to navigate and control any macOS application by bridging the gap between vision and action. By capturing high-resolution screenshots and processing them through the native Apple Vision framework, it identifies clickable text elements and generates a coordinate-mapped interface. This allows for precise interaction with legacy apps, web browsers, and complex desktop software that lacks a traditional API.
Integrating this into Openclaw Skills allows developers to build agents that perform multi-step workflows across different applications, such as extracting data from a spreadsheet and entering it into a custom CRM, or managing social media through official desktop clients. It provides a robust alternative to API-based automation by relying on the same visual cues a human user would follow.
Mac Use Use Cases
- Automating repetitive tasks in desktop apps that lack API support.
- Testing macOS application interfaces through automated visual regression.
- Creating cross-application workflows like data entry between legacy software and modern web apps.
- Navigating complex mini-programs within apps like WeChat or Slack.
- Remote assistance and automated troubleshooting of macOS system settings.
How Mac Use Works
- The skill identifies the target application window and brings it to the foreground using system-level commands.
- A screenshot is captured and processed using Apple Vision OCR to detect all visible text elements.
- An annotated image is generated with numbered markers, providing a visual map for the AI agent to reference.
- The agent selects an element by its assigned number or specific canvas coordinates based on the generated JSON metadata.
- Actions like clicknum, type, or scroll are executed to interact with the GUI directly.
- A follow-up screenshot is taken to verify the success of the action and prepare for the next step in the loop.
Mac Use Setup
Ensure you are running on macOS with Python 3 installed. You can install the necessary dependencies using the following commands:
# Install Python if not already present
brew install python
# Install required Python packages from the skill directory
pip3 install --break-system-packages -r requirements.txt
Mac Use Data Schema & Taxonomy
The skill utilizes a standardized 1000x1000 coordinate system to map application windows, ensuring consistent interaction regardless of screen resolution.
| Item | Description |
|---|---|
/tmp/mac_use.png |
The latest annotated screenshot showing numbered elements with green bounding boxes. |
/tmp/mac_use_elements.json |
A JSON map containing element numbers, text content, and canvas coordinates. |
at: [x, y] |
The normalized center point of an element on the 1000x1000 canvas. |
list output |
A JSON array of all visible windows including IDs, titles, and dimensions. |
Mac Use Advanced Features
- Multi-window handling with specific window IDs for targeting background processes or popups.
- Support for complex key combinations like cmd+shift+s for advanced application shortcuts.
- Native CJK and Unicode support for typing via clipboard paste, ensuring international character reliability.
- Fuzzy app name matching for quick window activation without needing exact titles.
- Seamless integration within the Openclaw Skills ecosystem for multi-agent coordinated desktop tasks.
SKILL.md
Loading
Related Openclaw Skills
Meta Ads Analytics & Event Finder
A conversational toolkit for monitoring Meta advertising performance, calculating key metrics, and identifying conversion events.
Zhihu AI Bot
A comprehensive automation tool for managing Zhihu Ring interactions, including content publishing, commenting, and engagement metrics.
paradiz
An automated VK interaction skill that calculates hotel stay costs from Excel price lists and manages the full booking lifecycle.
Breeze x402 Payment API
A payment-gated API skill for interacting with the Breeze Solana yield aggregator using x402 USDC micropayments.
Twitter Web AutoPoster
A specialized automation skill that enables AI agents to scrape trends, draft tweets, and publish posts directly to Twitter/X using browser-based interactions.
ZeeLin Twitter/X AutoPost
A browser-based automation tool for X (Twitter) that enables automated posting and follower management without requiring an official API key.
METADATA
Featured*
