A high-performance content extraction tool that converts web pages into clean, readable Markdown or plain text for AI processing.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install webscraper
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install webscraper using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is WebScraper?
The WebScraper skill is a specialized utility designed for Openclaw Skills that enables AI agents to read and interpret web content without the overhead of a full browser. It focuses on transforming raw HTML into structured, readable formats like Markdown or text, making it an ideal choice for processing articles, documentation, and blog posts.
By prioritizing speed and simplicity, this skill allows developers to integrate web-reading capabilities into their workflows using standard tools like curl and Node.js. It effectively bridges the gap between raw web data and AI-ready information while maintaining a lightweight footprint compared to heavy browser automation tools.
WebScraper Use Cases
- Summarizing long-form articles or blog posts by extracting the primary body text.
- Fetching technical documentation or API tutorials for real-time coding assistance.
- Scraping product details, pricing, and specifications from e-commerce sites for comparison.
- Converting news content into clean Markdown for archival or knowledge base integration.
- Extracting metadata, such as page titles and descriptions, for SEO analysis within Openclaw Skills.
How WebScraper Works
- The skill receives a target URL and optional parameters like extractMode (Markdown/Text) and maxChars.
- It initiates a request using the web_fetch tool or fallback methods like curl with customized User-Agents to avoid basic bot detection.
- Raw HTML is processed to strip away non-essential elements such as navigation bars, footers, and advertisements.
- The remaining content is parsed and formatted into the requested output type.
- The final structured response is returned, complete with source attribution and timestamps for the AI agent to utilize.
WebScraper Setup
To use this skill within the Openclaw Skills ecosystem, ensure your environment has the necessary binaries installed.
# Ensure core dependencies are available
sudo apt-get install curl
# Optional: Install Node.js dependencies for advanced parsing
npm install -g cheerio readability-cli
No complex configuration is required as the skill leverages the built-in web_fetch tool by default.
WebScraper Data Schema & Taxonomy
WebScraper organizes extracted data into a standardized response format to ensure consistency across different Openclaw Skills implementations.
| Field | Description | Type |
|---|---|---|
| Page Title | The extracted <title> or main <h1> of the page |
String |
| Source | The original URL of the content | String (URL) |
| Fetched | The ISO timestamp of the retrieval | Date |
| Content | The cleaned Markdown or plain text body | String |
| Summary | An optional 1-2 sentence overview of the page | String |
WebScraper Advanced Features
- Multi-mode extraction supporting both Markdown for rich context and plain text for token efficiency.
- Intelligent fallback mechanisms using curl and Node.js (Cheerio) when standard tools encounter specific site structures.
- Customizable User-Agent strings to mimic various browsers and devices, reducing the likelihood of being blocked.
- Built-in content truncation via maxChars to prevent context window overflow in LLMs.
- Rate-limiting best practices and error handling for robust batch processing tasks.
SKILL.md
Loading
Related Openclaw Skills
Subtitle Translator
A local, high-performance reference tool for subtitle translation patterns, debugging, and implementation best practices.
API Page Generator
A specialized skill for building, auditing, and optimizing API introduction and overview pages to bridge the gap between marketing and technical documentation.
RTK (Rust Token Kit)
A high-performance Rust-based proxy that compresses command-line output to drastically reduce LLM token costs.
Anthropic Chat Skill
A high-performance bridge for executing natural language tasks directly through the Anthropic Messages API using Claude models.
Customer Segmentation
A strategic marketing tool that categorizes customer bases into distinct groups based on behavioral, demographic, and value-based characteristics.
Fluid Network Solver & Reliability Analyzer
A specialized tool for calculating steady-state incompressible fluid network pressures, flow rates, and functional reliability across multiple scenarios.
Featured*
