A high-performance web scraping and crawling CLI that provides LLM-optimized markdown for AI agents.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install anycrawl-cli
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install anycrawl-cli using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is AnyCrawl?
AnyCrawl is a versatile CLI tool designed to bridge the gap between the web and AI agents. It enables seamless searching, scraping, and crawling of websites, converting complex HTML into clean, context-aware markdown.
By using the playwright engine by default, it handles modern web architectures effectively, making it a vital component for developers building Openclaw Skills that require real-time data or bulk content extraction. The tool is specifically tuned to output markdown that fits efficiently within LLM context windows.
AnyCrawl Use Cases
- Researching complex topics and answering questions when no specific URL is provided.
- Extracting clean markdown content from a known URL for deep LLM analysis.
- Mapping out a website's structure to locate specific subpages or hidden resources.
- Performing bulk content extraction from entire site sections, such as technical documentation or blog archives.
How AnyCrawl Works
- The agent identifies a need for web data through search, scrape, map, or crawl triggers based on user intent.
- A command is executed via the CLI using either anycrawl or npx anycrawl.
- The tool utilizes the playwright engine to render the target page and extract the relevant content.
- Raw HTML is transformed into LLM-optimized markdown to preserve structural context while minimizing token usage.
- Results are stored locally in the .anycrawl/ directory for incremental access, ensuring the agent doesn't perform redundant fetches.
AnyCrawl Setup
To get started with this skill for Openclaw Skills, ensure the CLI is installed and authenticated.
Run the login command:
anycrawl login
Alternatively, you can set your API key as an environment variable in your shell:
export ANYCRAWL_API_KEY='your_api_key_here'
It is recommended to add the output directory to your version control ignore file:
echo ".anycrawl/" >> .gitignore
AnyCrawl Data Schema & Taxonomy
AnyCrawl organizes its output in a structured format within a dedicated local directory to optimize performance and prevent redundant network requests.
| Feature | Description |
|---|---|
| Storage Directory | .anycrawl/ |
| File Format | LLM-optimized Markdown (.md) |
| Command Options | Use -o to specify custom output paths |
| Data Handling | Optimized for grep and incremental reads |
AnyCrawl Advanced Features
- Integrated search --scrape functionality to find and fetch full page content in a single operation.
- Intelligent mapping tools to discover deep URLs within a domain without full crawling.
- Bulk crawling capabilities designed for extracting specific site sections like /docs/ or /blog/.
- Native Playwright support for handling JavaScript-heavy and dynamic websites.
- Built-in efficiency filters to avoid redundant fetches by checking local caches first.
SKILL.md
Loading
Related Openclaw Skills
Summarize
A high-performance command-line utility for generating AI-driven summaries of web content, documents, and multimedia.
Shopify Product Description Generator
A conversion-focused copywriting engine that turns raw product data into Shopify-ready PDP descriptions.
Laundry Tracker
A lightweight CLI utility for tracking laundry loads, costs, and maintenance schedules directly through an AI agent.
48h-Expert-Protocol (Cognitive-Compressor)
A high-authority cognitive extraction system that serializes complex domain expertise into structured mental models for AI agents.
Oven
A robust command-line utility for tracking oven usage, cooking schedules, and household maintenance tasks.
Issue Request Manager
A comprehensive AI-driven tool for managing the entire lifecycle of issue requests across platforms like GitHub, GitLab, and Jira.
Featured*
