A powerful web scraping skill for AI agents that leverages local Crawl4AI instances to extract full-page content with JavaScript rendering.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install crawl-for-ai
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install crawl-for-ai using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Crawl4AI Web Scraper?
The Crawl4AI Web Scraper is a robust tool designed for high-fidelity web data extraction. By interfacing with a local Crawl4AI instance, this skill allows AI agents to bypass the limitations of standard scrapers that struggle with JavaScript-heavy websites. It provides two distinct modes: a simplified proxy for clean markdown consumption and a direct mode for comprehensive data retrieval including links, media, and tables.
Because this skill operates on your own infrastructure, it offers unlimited usage and significant cost savings compared to external search APIs like Tavily. It is an essential component for developers building sophisticated agents within the Openclaw Skills ecosystem, ensuring that your agents have access to the most accurate and complete web data available.
Crawl4AI Web Scraper Use Cases
- Scraping dynamic Single Page Applications (SPAs) that require full JavaScript execution to reveal content.
- Gathering comprehensive datasets including images, links, and structured tables for deep-dive research tasks.
- Implementing cost-effective, high-volume web crawling without hitting third-party API rate limits or incurring extra costs.
- Feeding perfectly formatted markdown into LLMs for accurate summarization of complex web pages.
How Crawl4AI Web Scraper Works
- The AI agent initiates a request via the provided CLI script, passing a target URL and optional flags.
- The skill script connects to the local Crawl4AI REST API using the configured CRAWL4AI_URL.
- Crawl4AI renders the target webpage, executing all necessary JavaScript to ensure dynamic content is visible.
- The raw site data is processed into either a clean markdown format or a detailed JSON structure depending on the input parameters.
- The refined data is returned to the agent, providing it with the necessary context to complete its task.
Crawl4AI Web Scraper Setup
First, ensure you have a running instance of Crawl4AI. Then, configure your environment variables to point to your instance:
export CRAWL4AI_URL="http://localhost:11235"
# Optional: Set an API key if your instance is secured
export CRAWL4AI_KEY="your_secret_key"
You can test the skill directly from your terminal:
# Standard markdown extraction
node scripts/crawl4ai.js "https://example.com"
# Extraction with full metadata and JSON structure
node scripts/crawl4ai.js "https://example.com" --json
Crawl4AI Web Scraper Data Schema & Taxonomy
The skill provides structured output tailored for AI consumption. Below is the metadata taxonomy used when requesting full data:
| Field | Type | Description |
|---|---|---|
| markdown | String | The core content of the page converted to clean markdown |
| html | String | The raw HTML structure of the rendered page |
| links | Array | A collection of all internal and external URLs found on the page |
| media | Array | Metadata for images, videos, and other assets found |
| metadata | Object | Includes page title, description, and other SEO tags |
Crawl4AI Web Scraper Advanced Features
- Dual-port architecture providing a simplified Proxy (11234) for OpenWebUI and a Direct (11235) endpoint for technical depth.
- Full support for JavaScript rendering, making it far more capable than standard request-based scrapers.
- Authenticated request handling via optional API key headers for secure self-hosted environments.
- Zero-throttle performance, allowing for rapid, unlimited scraping as part of the Openclaw Skills framework.
- Flexible output options allowing users to toggle between raw content and structured JSON data.
SKILL.md
Loading
Related Openclaw Skills
Cozi Family Organizer
A powerful integration that allows AI agents to interact directly with the Cozi Family Organizer for managing shared schedules and lists.
Agent Earth
Agent Earth allows AI agents to perform virtual city walks by researching landmarks, generating rich waypoint data, and publishing results to a global platform.
Knowledge Forge
Knowledge Forge transforms raw experience into transferable cognitive assets and structured models that others can immediately apply.
Smoobu Property Management
An automation skill for managing Smoobu vacation rental properties, reservations, and real-time availability across multiple booking channels.
Mealie
A powerful API client for Mealie that lets you manage recipes, meal plans, and shopping lists through AI-driven workflows.
PMC Harvest
A specialized utility for searching and harvesting full-text scientific articles and metadata from PubMed Central using official NCBI APIs.
Featured*
