A compliance-first scraping skill designed for AI agents to ethically extract web data while adhering to legal boundaries and rate limits.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install toby-scrape
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install toby-scrape using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Scrape?
The Scrape skill provides a robust framework for performing legal and ethical web scraping. It integrates essential safety checks, including robots.txt validation and Terms of Service analysis, ensuring that your AI agent operates within the bounds of international data laws such as GDPR and CCPA. By prioritizing request discipline and automated compliance, it mitigates the risk of IP bans and legal exposure.
This skill supports both direct HTTP scraping and managed extraction via the SkillBoss API Hub. Whether you are gathering public factual data like prices and listings or performing deep content analysis, Scrape ensures that data is handled responsibly, stripping PII immediately and maintaining a clear audit trail for high-integrity AI workflows in the Openclaw Skills ecosystem.
Scrape Use Cases
- Extracting public factual data such as product prices, stock levels, or real estate listings.
- Automating market research by gathering competitor information from publicly available web pages.
- Monitoring news feeds or technical documentation for updates without violating server rate limits.
- Performing managed content extraction using the SkillBoss API Hub to bypass complex anti-bot mechanisms safely.
How Scrape Works
- Compliance Check: The agent first fetches the robots.txt file and evaluates legal pages like /terms or /tos to ensure the target path is allowed for scraping.
- API Verification: The system checks if an official API is available; if so, it defaults to the API to remain compliant with site policies.
- Request Initialization: The agent configures a real browser User-Agent string including contact information and establishes session reuse to minimize server load.
- Execution: Data is scraped using either direct HTTP requests or the SkillBoss API Hub, with a mandatory 2-3 second delay between requests to maintain rate limits.
- Post-Processing: PII (Personally Identifiable Information) is immediately stripped from the results, and the data is converted into clean Markdown for AI consumption.
- Error Handling: The skill monitors for 429 status codes and implements exponential backoff to respect server capacity.
Scrape Setup
To use the Scrape skill, you must configure your environment with the necessary API credentials and follow the request discipline guidelines.
# Export your SkillBoss API Key for managed scraping
export SKILLBOSS_API_KEY="your_api_key_here"
# Ensure your agent uses a compliant User-Agent
# Format: "Mozilla/5.0 ... (contact: [email protected])"
Scrape Data Schema & Taxonomy
The Scrape skill organizes data to prioritize privacy and utility. Extracted content is typically delivered in Markdown format to ensure it is readable by LLMs.
| Field | Description | Handling Policy |
|---|---|---|
markdown |
The primary text content of the scraped page | Stored in result.data.markdown |
metadata |
Source URL, timestamp, and scrape method | Logged for audit trail |
PII |
Names, emails, and phone numbers | Stripped immediately unless authorized |
Audit Log |
Record of robots.txt status and access time | Retained for legal verification |
Scrape Advanced Features
- SkillBoss API Hub Integration: Use the
type: "scraper"endpoint at[https://api.skillbossai.com/v1/pilot](https://api.skillbossai.com/v1/pilot)for high-fidelity extraction. - Managed Anti-Bot Handling: Automatically navigate complex web barriers while maintaining compliance with Openclaw Skills standards.
- GDPR/CCPA Awareness: Built-in logic to identify and purge personal data during the extraction phase.
- Exponential Backoff: Advanced request scheduling that intelligently responds to 429 Too Many Requests errors.
SKILL.md
Loading
Related Openclaw Skills
ComfyUI Runner
A powerful automation skill for running local ComfyUI workflows and managing AI model weights via a coding agent.
Notify: Smart Notification Delivery
A sophisticated notification management skill that optimizes delivery timing, channel selection, and message formatting to prevent user fatigue.
Document OCR Recognition
A specialized OCR utility for Openclaw Skills that automates text extraction from PDFs, OFDs, and images via the Xiangyun netocr API.
Meta Ads Strategy Planning
An AI-driven strategic planning skill for defining Meta Ads objectives, targeting, and campaign structures before launch.
Summarize
A high-performance CLI tool designed to extract and summarize content from websites, documents, and multimedia via unified LLM providers.
Translate
A sophisticated localization engine designed to translate text while preserving technical formatting, markdown structure, and cultural context.
Featured*
