An intelligent web scraping tool that extracts clean main body content and converts it to Markdown for AI processing.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install scrapling-web-fetch
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install scrapling-web-fetch using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Scrapling Web Fetch?
Scrapling Web Fetch is a specialized tool designed to solve the challenges of modern web data extraction. By combining the power of the Scrapling library with html2text, this skill intelligently identifies the core content of a webpage—such as blog posts, news articles, or announcements—while stripping away irrelevant noise like navigation bars, ads, and footers. It is a vital component for developers looking to enhance their collection of Openclaw Skills with robust, high-fidelity data retrieval capabilities.
One of the standout features of this skill is its native support for WeChat official account articles, including specialized cleaning of platform-specific noise. By transforming complex HTML into clean, structured Markdown, it ensures that your AI agents receive only the most relevant information, significantly reducing token consumption and improving the performance of summarization and research workflows.
Scrapling Web Fetch Use Cases
- Extracting main article text from blogs, news portals, and corporate announcements.
- Scraping WeChat articles with dedicated noise cleaning and footer removal.
- Converting modern, dynamically rendered web pages into clean Markdown for LLM analysis.
- Enhancing data quality for AI agents where standard fetch methods fail due to anti-scraping or complex DOM structures.
How Scrapling Web Fetch Works
- The skill initiates the extraction process via a Python script, accepting a target URL and a character limit.
- It employs a prioritized selector hierarchy, first attempting to locate the content within
article,main, or specific content-heavy CSS classes. - Upon identifying the main body, the HTML is processed through the html2text engine to produce a readable Markdown format.
- If no high-priority containers are found, the system falls back to the
bodytag to ensure no content is missed. - The final output is truncated to the specified character length to maintain compatibility with model context windows.
Scrapling Web Fetch Setup
To use this within your environment, ensure you have the necessary Python dependencies installed:
python3 -m pip install scrapling html2text
You can then run the fetcher directly from the CLI:
python3 scripts/scrapling_fetch.py <url> <max_chars>
Scrapling Web Fetch Data Schema & Taxonomy
The skill focuses on clean text output but supports structured variations. Metadata about the extraction process is often sent to stderr for debugging.
| Input Parameter | Type | Description |
|---|---|---|
url |
string | The target URL to be scraped. |
max_chars |
integer | The maximum number of characters to return. |
--json |
flag | Optional flag to return a structured JSON response instead of raw Markdown. |
Scrapling Web Fetch Advanced Features
- Intelligent WeChat Article Support: Automatically cleans platform-specific noise and redundant footers unique to WeChat.
- Priority Selector Logic: Uses a sophisticated fallback chain (article > main > .post-content) to find the 'meat' of any page.
- Token Optimization: Focuses strictly on main body content to maximize the efficiency of Openclaw Skills in AI workflows.
- Debug Mode: Provides detailed stderr output to identify which CSS selector successfully captured the content.
SKILL.md
Loading
Related Openclaw Skills
Scrapling Web Fetch
A high-performance web content extractor that transforms messy HTML into clean, LLM-ready Markdown while removing noise and reducing token usage.
Prepper Survival & Emergency AI Skill
A specialized survival and emergency preparedness skill that leverages the dolphin-llama3 model via Ollama to provide uncensored, practical knowledge.
FileWave UEM API Skill
A powerful CLI skill to query and manage FileWave UEM device inventory, supporting multi-server environments and fleet analytics.
Clawtter
A social networking skill for AI agents to post updates, engage with the community, and manage an autonomous online presence.
Baidu Baike
A specialized knowledge retrieval tool that enables AI agents to query and extract authoritative encyclopedia data from Baidu Baike.
TaskMaster
An intelligent project management system that breaks down complex goals and orchestrates AI sub-agents using cost-effective model selection.
Featured*
