A secure extraction tool designed to transform messy public web pages into clean, structured, and reusable local data.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install scraper
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install scraper using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Scraper?
The Scraper skill is a specialized utility within the Openclaw Skills ecosystem designed for safe and structured data extraction. It empowers AI agents to navigate public or user-authorized URLs, fetch page content, and strip away boilerplate noise to reveal high-value readable text.
By focusing on local-only output and adhering to strict safety boundaries, this skill ensures that data processing remains private and compliant. It is an essential component for developers building workflows that require automated content collection, summarization, or the transformation of web-based information into analysis-ready formats.
Scraper Use Cases
- Extracting clean text from public articles or documentation for AI-driven summarization.
- Automating the collection of public data points into local JSON or text files.
- Transforming messy HTML content into readable formats for RAG (Retrieval-Augmented Generation) pipelines.
- Maintaining a local history of web research and scraping jobs for auditability.
How Scraper Works
- The agent receives a public URL and initializes the local storage environment if it doesn't already exist.
- The script fetches the page content using standard headers to ensure a clean download.
- An extraction process converts the raw HTML into cleaned, plain text by removing scripts, styles, and boilerplate elements.
- The cleaned content is saved to a unique file in the local workspace directory.
- The job is registered in a central JSON manifest to allow for easy retrieval and listing of past activities.
Scraper Setup
To integrate this skill into your Openclaw Skills environment, ensure you have Python 3 installed as python3. No external third-party packages are required.
Initialize the storage directory before the first run:
python3 init_storage.py
You can then fetch and extract content immediately:
python3 fetch_page.py --url "https://example.com"
python3 extract_text.py --url "https://example.com"
Scraper Data Schema & Taxonomy
The skill maintains a structured local file system to ensure data persistence and organization:
| Path | Purpose |
|---|---|
~/.openclaw/workspace/memory/scraper/jobs.json |
A master manifest file tracking all scraping metadata and job history. |
~/.openclaw/workspace/memory/scraper/output/ |
Directory containing the individual cleaned text files and raw extractions. |
Each entry in the jobs.json file typically includes the source URL, timestamp, and a reference to the local file path.
Scraper Advanced Features
- Local-first architecture ensuring that extracted data never leaves your controlled environment.
- Automated boilerplate stripping to optimize token usage for downstream LLM processing.
- Built-in job management system to track and list previous extraction tasks.
- Seamless integration with other Openclaw Skills for multi-stage data transformation pipelines.
SKILL.md
Loading
Related Openclaw Skills
Hotel Decision Engine
A local-first decision engine for comparing hotels, managing trip stay tradeoffs, and reaching booking readiness without external APIs.
Jisu Unified API Integration
A unified data gateway that enables AI agents to access over 100 diverse APIs from JisuAPI using a single configuration and authentication key.
Skill Creator
The Skill Creator provides a structured framework and automated toolkit for developing, validating, and packaging specialized modular extensions for AI agents.
Answer Box
Answer Box is a healing Openclaw Skills tool that provides instant, positive guidance for daily dilemmas through simple silent reflection.
Giggle Music
An advanced AI music generation skill for Openclaw Skills that transforms text descriptions and lyrics into professional audio tracks.
Requirement Review Skill
A professional AI evaluation tool designed to perform deep-focus reviews of PRDs and requirement documents to identify systemic risks and root causes.
Featured*
