A smart extraction engine that automatically detects platforms to download articles or high-quality videos with ease.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install web-fetcher
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install web-fetcher using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Web Fetcher?
Web Fetcher is a versatile content retrieval tool designed to integrate seamlessly with Openclaw Skills. It solves the challenge of scraping fragmented or protected web content by employing platform-specific strategies for WeChat, Feishu, Zhihu, Bilibili, and more. Whether you need to archive documentation or fetch media for research, this skill provides a unified interface for complex data gathering.
By utilizing specialized libraries like scrapling for high-speed HTTP fetching and camoufox for stealth browser automation, Web Fetcher ensures high success rates even on sites with aggressive anti-bot measures. It simplifies the developer workflow by handling lazy-loaded images, virtual scrolling in documents, and authenticated media downloads automatically.
Web Fetcher Use Cases
- Archiving WeChat official account articles for local knowledge bases or RAG applications.
- Batch downloading research videos from Bilibili, YouTube, or Douyin for offline analysis.
- Exporting complex, dynamic Feishu documents into structured Markdown files while preserving image links.
- Scrapping content from JS-heavy platforms like Xiaohongshu and Weibo using advanced stealth browser techniques.
How Web Fetcher Works
- The user provides a URL or a file containing multiple links to the Web Fetcher script.
- The internal router analyzes the URL to identify the source platform and the required extraction method.
- For articles, the system selects either a fast HTTP parser or a stealth browser based on the site's complexity.
- For videos, the skill leverages yt-dlp to negotiate streams and download the highest available quality.
- If the platform requires specialized handling (like Feishu's virtual scroll), the skill executes custom logic to collect all content blocks.
- Extracted content is converted to Markdown or saved as media files in the designated output directory.
Web Fetcher Setup
Install the core dependencies required for Openclaw Skills functionality:
# Install basic fetching and processing libraries
pip install scrapling yt-dlp html2text
# Install the stealth browser for protected sites
pip install camoufox && python3 -m camoufox fetch
Run the fetcher directly from your terminal:
python3 fetcher.py "https://mp.weixin.qq.com/s/example" -o ./downloads
Web Fetcher Data Schema & Taxonomy
The skill organizes extracted data into structured files. Metadata is handled internally to ensure consistent routing.
| Component | Format | Description |
|---|---|---|
| Articles | .md | Extracted text converted to Markdown with image references. |
| Videos | .mp4 / .webm | Media files downloaded at specified quality (default 1080p). |
| Audio | .mp3 / .m4a | Audio-only extraction when the --audio-only flag is used. |
| URL Lists | .txt | Input file format for batch processing, supporting comments with #. |
Web Fetcher Advanced Features
- Multi-platform smart routing that automatically chooses between scrapling, camoufox, or yt-dlp.
- Stealth browsing capabilities to bypass detection on platforms like Weibo and Xiaohongshu.
- Authenticated fetching using --cookies-browser to access member-only content or private documents.
- Virtual scroll handling for document platforms that load content dynamically as the user scrolls.
- Customizable CSS selectors for precision extraction on unsupported or generic websites.
SKILL.md
Loading
Related Openclaw Skills
ProtoHub AI Agent Integration
An automated utility for AI agents to package, publish, and manage web prototypes on the ProtoHub platform.
Aqara Open API Skill
A professional-grade interface for controlling Aqara smart home hardware and managing spatial hierarchies via the Aqara Open Platform API.
Polymarket Alpha Suite
A suite of 6 institutional-grade trading tools for Polymarket prediction markets featuring arbitrage, scalping, and alpha detection algorithms.
Square Post
A specialized AI agent skill designed to automate text-based posting to the Binance Square social platform.
Agent Control
A command-driven interface for orchestrating isolated OpenClaw agents directly through chat shorthand.
Campaign Analytics
A production-grade suite for analyzing marketing campaign performance through attribution modeling, funnel conversion, and ROI calculation.
Featured*
