High-performance web scraping tools designed to fetch, extract, and clean data from web pages for AI agent workflows.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install xpr-web-scraping
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install xpr-web-scraping using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Web Scraping?
The Web Scraping skill provides a robust suite of tools designed to bridge the gap between AI agents and the live web. By enabling agents to fetch URLs, extract structured data, and navigate complex link hierarchies, this skill transforms how agents gather information. It is a vital component of the Openclaw Skills ecosystem, focusing on delivering clean, LLM-ready content while maintaining performance and reliability.
This skill handles the heavy lifting of HTML parsing, deduplication, and link resolution. Whether you need to strip a page down to its core text for efficient token usage or preserve structural formatting with markdown, these tools offer the flexibility required for professional-grade web research and automation.
Web Scraping Use Cases
- Extracting clean, readable text from blog posts or documentation for LLM analysis.
- Automated discovery of specific file types (like PDFs) across a website using regex link filtering.
- Performing competitive research by scraping up to 10 target URLs simultaneously.
- Archiving web-based content into a persistent data store for audit trails and job evidence.
How Web Scraping Works
- The AI agent invokes a tool like scrape_url or scrape_multiple based on the research objective.
- The system fetches the target HTML and processes it based on the specified format (text, markdown, or HTML).
- Metadata such as page titles and descriptions are extracted to provide context to the scraped content.
- Link discovery tools resolve relative paths to absolute URLs and deduplicate results for a clean output.
- Processed data is returned to the agent, ready for synthesis or storage.
Web Scraping Setup
To utilize this skill, ensure your agent environment is configured to support the Openclaw Skills scraping library. No external API keys are required for standard operation.
# List available skills to confirm web-scraping is active
openclaw skills list
Web Scraping Data Schema & Taxonomy
| Component | Description | Data Type |
|---|---|---|
| Content | The cleaned text or formatted markdown from the page | String |
| Metadata | Title, description, and link counts for the URL | Object |
| Links | Array of absolute URLs, anchor text, and link types | Array |
| Format | The output structure (text, markdown, or html) | Enum |
Web Scraping Advanced Features
- Parallel Execution: Fetch up to 10 URLs at once using Promise.allSettled logic to ensure one failure doesn't stop the job.
- Regex Pattern Matching: Target specific resources during link extraction using the pattern parameter for precise filtering.
- Smart Content Cleaning: Automatically strips HTML noise to provide high-density text for AI consumption.
- Domain Throttling: Built-in best practices to manage rate limits and prevent IP blocking during extensive research tasks.
SKILL.md
Loading
Related Openclaw Skills
Structured Data
A comprehensive toolkit for parsing CSV datasets, converting JSON to CSV, and generating professional SVG visualizations.
XPR Network NFT Lifecycle Manager
A comprehensive toolset for managing the full AtomicAssets and AtomicMarket NFT lifecycle on the XPR Network.
LOAN Protocol Lending & Borrowing
An AI agent skill for interacting with the LOAN Protocol on XPR Network to supply assets, borrow against collateral, and manage DeFi positions.
Code Sandbox Skill
A secure, sandboxed environment for executing JavaScript code and mathematical expressions within AI agent workflows.
ChilledSites
An AI-driven website builder that allows coding agents to generate, edit, and deploy professional landing pages and portfolios using simple text prompts.
Hybrid Training Plan
A professional fitness integration for Openclaw Skills that allows users to manage their training plans and log workouts via natural language.
Featured*
