A specialized framework for ethical web scraping that prioritizes robots.txt compliance, rate limiting, and data privacy laws.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install scrape
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install scrape using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Scrape?
The Scrape skill is a comprehensive toolset designed for developers who need to extract web data while maintaining strict legal and ethical standards. It functions as a core component of Openclaw Skills, providing a structured approach to identifying public data, respecting server boundaries, and adhering to modern legal precedents like hiQ v. LinkedIn.
By automating the checklist for robots.txt and Terms of Service (ToS) verification, this skill reduces the risk of CFAA violations and copyright infringement. It focuses on high-integrity data gathering, ensuring that every request is disciplined and every piece of data collected is handled with privacy-first principles.
Scrape Use Cases
- Monitoring public factual data such as real-time pricing and inventory listings.
- Performing competitive market research without violating robots.txt protocols.
- Collecting datasets for training AI models while ensuring PII is stripped automatically.
- Auditing web properties for compliance with data protection laws like GDPR and CCPA.
How Scrape Works
- Executes a pre-scrape compliance check by fetching and parsing the target domain's robots.txt file.
- Analyzes legal documents like /terms and /tos to verify scraping permissions.
- Checks for the availability of official APIs to prioritize authorized data access over scraping.
- Initiates a disciplined request lifecycle with a mandatory 2-3 second delay between fetches to prevent server strain.
- Handles HTTP 429 status codes using exponential backoff to demonstrate good faith and technical discipline.
- Filters all incoming data to remove personally identifiable information before storage.
Scrape Setup
To integrate this capability into your Openclaw Skills workflow, use the following configuration steps:
# Add the scrape skill to your agent configuration
openclaw install scrape
# Set your mandatory User-Agent contact information
openclaw config scrape --user-agent "Mozilla/5.0 (contact: [email protected])"
Scrape Data Schema & Taxonomy
The skill organizes extracted data and metadata to maintain an audit trail for legal protection:
| Attribute | Description | Data Type |
|---|---|---|
source_domain |
The root domain being accessed for data extraction. | String |
compliance_status |
Verification result of robots.txt and ToS checks. | Boolean |
request_log |
A detailed log of timestamps and status codes (e.g., 200, 429). | Array |
sanitized_payload |
The final extracted data with all PII stripped. | JSON |
audit_trail |
Evidence of good faith including User-Agent used and delay settings. | Object |
Scrape Advanced Features
- Intelligent robots.txt parser with automated path restriction enforcement.
- Session reuse protocols to minimize server-side load and overhead.
- Automated PII (Personally Identifiable Information) detection and immediate sanitization.
- Support for exponential backoff logic when encountering rate limits or 429 errors.
- Integration with Openclaw Skills audit logging for long-term compliance tracking.
SKILL.md
Loading
Related Openclaw Skills
SciPy
A comprehensive scientific computing toolkit for solving optimization, statistics, signal processing, and linear algebra problems with production-ready code.
Science
A versatile skill that enables AI agents to calibrate scientific communication and research support based on the user's specific level of expertise.
School
A comprehensive AI-driven educational framework for K-12 students featuring adaptive learning, exam preparation, and robust parental oversight.
Scholar
A professional academic research skill that enables AI agents to find, filter, and analyze peer-reviewed literature and citations.
Screenshots
A comprehensive automation skill for generating professional, device-framed, and marketing-ready mobile app screenshots for various app stores.
Search Engine
A comprehensive framework for designing and building production-grade search engines with optimized indexing, retrieval logic, and relevance controls.
Featured*
