Webclaw is an advanced web scraping, crawling, and research tool optimized for LLMs that easily bypasses bot protections and parses dynamic JavaScript sites.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install webclaw-extraction
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install webclaw-extraction using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Webclaw?
Webclaw is a state-of-the-art web extraction engine designed specifically for large language models and autonomous agents. As one of the most powerful Openclaw Skills, it acts as a robust replacement for basic web fetch engines, effortlessly handling sites with aggressive bot protections, CAPTCHAs, or heavy client-side JavaScript rendering.
By converting raw web pages into clean, structured Markdown, text, or LLM-optimized JSON, Webclaw allows developers to feed high-fidelity data directly to their agents. It features a rich suite of tools including deep crawling, site mapping, visual branding extraction, change detection, and specialized vertical scrapers for major platforms.
Webclaw Use Cases
- When standard fetch tools return empty results or 403 Forbidden errors due to bot-detection software.
- Bypassing JavaScript rendering steps to scrape single-page applications (SPAs) natively.
- Extracting clean, structured JSON schemas (pricing tables, product catalogues, reviews) without writing custom parsers.
- Gaining insight into target backends by scanning frontend Javascript bundles for hidden API, GraphQL, or WebSocket endpoints.
- Setting up real-time web monitoring and change notification triggers via cron jobs and webhooks.
- Conducting deep multi-source academic or market research automatically with agentic loops.
How Webclaw Works
- Inbound Request: The agent or application initiates a scrape, crawl, or map request to the Webclaw API using the standard client or Openclaw Skills configuration.
- Route Assessment: Webclaw evaluates the target URL. If it matches one of its 28 vertical extractor patterns (like GitHub, Reddit, ArXiv, Amazon), it instantly routes the request to a dedicated high-efficiency extractor.
- Bot Bypass & Rendering: If the URL is generic, Webclaw passes the request through its automatic bot-protection bypass and headless browser system to handle JavaScript hydration.
- Clean up & Formatting: The 20-step optimization pipeline strips out unnecessary markup, sidebars, and trackers to construct clean outputs in Markdown, plain text, JSON, or an LLM-optimized representation.
- Response Delivery: The structured payload, complete with metadata and caching parameters, is delivered back to the calling process.
Webclaw Setup
Set up Webclaw as part of your toolkit for Openclaw Skills by installing the CLI tool and configuring your credentials.
Installation
npx create-webclaw
Configuration
Webclaw requires an API key to access cloud features, such as bot protection bypasses and deep research. Set your environment variable:
export WEBCLAW_API_KEY="your_api_key_here"
Webclaw Data Schema & Taxonomy
Webclaw structures extraction results into clear, parseable schemas depending on the endpoint chosen. Below is the JSON response structure for a standard single-page Scrape request:
| Field | Type | Description |
|---|---|---|
url |
string | The requested web page address. |
metadata |
object | Contains title, page description, original language, and word count. |
markdown |
string | Cleaned Markdown extraction of the main article content. |
llm |
string | Optimized format with URL references specifically for AI models. |
antibot |
object | Diagnostics indicating if anti-bot bypass was triggered and execution time. |
youtube |
object | Optional metadata block populated only during YouTube video scraping (includes transcripts). |
For custom data extractions using schemas, Webclaw guarantees compliance with the structured JSON schema provided in your POST request body.
Webclaw Advanced Features
- Firecrawl v2 Compatibility Layer: Drop-in endpoint replacement allowing existing systems using Firecrawl to migrate seamlessly to Webclaw.
- Deep Multi-Source Research: Autonomous agentic iterations (search-read-analyze cycles) that synthesize comprehensive reports from up to 100 sources.
- Local-First CLI Wrapper: A hybrid approach executing free, local fetches for standard HTML pages and falling back to Cloud API only when bot detection is triggered.
- JavaScript Endpoint Discovery: Scans inline scripts and source bundles to automatically map relative paths, WebSockets, and GraphQL API routes.
- Visual Identity Analysis: Extracts site brand identifiers like font family usage, logo vectors, favicons, and primary hexadecimal colors.
SKILL.md
Loading
Related Openclaw Skills
Affiliate Content Writer
An AI-powered affiliate content writer optimized to create high-conversion reviews, guides, and listicles for JD Union, Taobao Alliance, and Duoduo Jinbao.
eBook & Template Factory
An AI agent skill designed to generate market-ready eBooks, templates, and digital products optimized for monetization on popular Chinese digital marketplaces.
Huawei Cloud SWR Image Management
An advanced container image repository management automation tool built on top of the hcloud CLI for Huawei Cloud SWR.
Huawei Cloud UCS Cluster Onboarding Manager
An enterprise-grade orchestration skill for managing the end-to-end lifecycle, fleet organization, and secure credential provisioning of Huawei Cloud UCS clusters via automated CLI execution.
Xianyu Virtual Goods Auto-Seller
An AI-powered toolkit designed to fully automate and optimize the creation, listing, and customer response workflows for virtual goods on the Xianyu marketplace.
Instagram Post Comments Extractor
A powerful Openclaw Skills capability designed to scrape paginated comments, usernames, timestamps, and reaction metrics from any Instagram post.
Featured*
