A high-performance web content extractor that transforms messy HTML into clean, LLM-ready Markdown while removing noise and reducing token usage.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install clean-web-fetch
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install clean-web-fetch using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Scrapling Web Fetch?
Scrapling Web Fetch is a specialized tool designed to solve the problem of messy web data extraction. By focusing on the core content of modern web pages, it strips away headers, footers, and sidebars to deliver pure, readable text. This skill is particularly effective for processing news articles, blog posts, and WeChat official accounts, ensuring that your AI agents receive only the most relevant information.
Built on the robust scrapling library, it handles dynamic rendering and anti-scraping measures more effectively than standard fetch methods. By converting HTML directly into Markdown, it preserves essential structure while significantly reducing token consumption for downstream LLM processing within Openclaw Skills.
Scrapling Web Fetch Use Cases
- Extracting main article content from news sites, blogs, and corporate announcements.
- Converting complex web pages into clean Markdown for efficient AI summarization.
- Scraping WeChat Official Account articles with automated noise and footer removal.
- Improving data reliability for pages that are difficult to fetch using standard HTTP requests due to dynamic rendering.
How Scrapling Web Fetch Works
- The skill executes the scrapling_fetch.py script with a target URL and character limit.
- It applies a prioritized selector strategy, searching for tags like article, main, or specific content classes to find the signal in the noise.
- Upon identifying the main body, it uses html2text to convert the HTML structure into clean Markdown.
- If no specific content container is found, it falls back to the full body tag to ensure data capture.
- The final output is truncated to the specified character limit to optimize for context window constraints.
Scrapling Web Fetch Setup
Ensure you have the necessary dependencies installed in your environment:
python3 -m pip install scrapling html2text
You can then run the skill using the following command structure within your Openclaw Skills workspace:
python3 scripts/scrapling_fetch.py <url> <max_chars>
Scrapling Web Fetch Data Schema & Taxonomy
The skill provides flexible output formats depending on the required integration level. By default, it returns a plain Markdown string. For programmatic use, the --json flag can be appended to receive a structured object.
| Field | Type | Description |
|---|---|---|
| content | string | The extracted Markdown text content |
| selector | string | The CSS selector that successfully matched the content |
| url | string | The source URL of the fetched page |
| status | integer | The HTTP response status code |
Scrapling Web Fetch Advanced Features
- Intelligent WeChat article cleanup including footer noise and metadata removal.
- Multi-tier fallback mechanism from specific article selectors to general body tags.
- Custom character limit enforcement to prevent token overflow in AI models.
- Debug mode via stderr to identify which CSS selector was successfully matched during extraction.
- Enhanced stability for modern web pages that usually fail with standard fetch tools in Openclaw Skills.
SKILL.md
Loading
Related Openclaw Skills
Prepper Survival & Emergency AI Skill
A specialized survival and emergency preparedness skill that leverages the dolphin-llama3 model via Ollama to provide uncensored, practical knowledge.
FileWave UEM API Skill
A powerful CLI skill to query and manage FileWave UEM device inventory, supporting multi-server environments and fleet analytics.
Clawtter
A social networking skill for AI agents to post updates, engage with the community, and manage an autonomous online presence.
Honcho Memory System
A production-grade memory pipeline that replaces standard embedding retrieval with reasoned, evolving understanding across multi-agent fleets.
Scrapling Web Fetch
An intelligent web scraping tool that extracts clean main body content and converts it to Markdown for AI processing.
Baidu Baike
A specialized knowledge retrieval tool that enables AI agents to query and extract authoritative encyclopedia data from Baidu Baike.
Featured*
