A sophisticated data extraction skill that combines lightweight HTML fetching with full browser automation to turn websites into structured JSON or CSV data.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install web-scraping
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install web-scraping using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Web Scraping?
The Web Scraping skill is a versatile tool within the Openclaw Skills library designed to bridge the gap between raw web content and structured intelligence. It intelligently selects between lightweight fetching for static pages and full browser orchestration for complex, modern web applications. Whether dealing with single-page applications (SPAs), login-gated content, or infinite scrolling interfaces, this skill provides a reliable framework for data harvesting.
By prioritizing the lightest reliable method first, the skill ensures efficiency while maintaining the capability to handle high-friction environments. It is engineered to handle dynamic DOM inspection, stateful interactions like filtering and searching, and the normalization of messy web data into clean, developer-ready schemas.
Web Scraping Use Cases
- Extracting structured product listings or pricing data from e-commerce platforms.
- Automating the collection of news articles, research papers, or blog posts for content aggregation.
- Monitoring website changes or updates by periodically scraping specific target fields.
- Turning unstructured dynamic webpage content into organized JSON or CSV files for data analysis.
- Navigating complex workflows such as paginated results, hidden tabs, or search-filtered lists.
How Web Scraping Works
- Target Identification: The agent identifies the target URL and defines the specific data fields required for the task.
- Method Selection: It chooses between web_fetch for static HTML or browser for dynamic, JavaScript-heavy sites.
- State Management: For dynamic sites, the browser opens the page and takes snapshots to inspect the DOM, performing interactions like clicking or scrolling as necessary.
- Extraction & Normalization: Relevant fields are extracted and converted into a stable, predefined schema to ensure data consistency.
- Serialization: If multiple pages are involved, the skill serializes requests to maintain reliability and avoid anti-bot triggers.
- Deduplication & Storage: Results are deduplicated by URL or ID and saved directly to the workspace in the requested format.
Web Scraping Setup
To enable this skill within your environment, ensure your agent has access to the core web tools. Installation typically follows the standard Openclaw Skills integration pattern:
# Verify your agent has the necessary browser drivers
openclaw install web-scraping
# Ensure dependencies for browser automation are met
npm install playwright # or equivalent driver used by the agent
Web Scraping Data Schema & Taxonomy
The skill organizes extracted data into structured formats to facilitate easy integration with other tools. Typical outputs follow these structures:
| Feature | Description |
|---|---|
| JSON Format | An array of objects containing explicit keys like title, url, source, and date. |
| CSV/TSV | Flat rows designed for spreadsheet exports and bulk data processing. |
| Checkpointing | For large-scale jobs, partial results are stored in workspace files to prevent data loss. |
| Naming Convention | Output files are stored under the local workspace with descriptive, task-specific filenames. |
Web Scraping Advanced Features
- Intelligent Failover: Automatically switches sources or reports blocks if a site prevents access.
- Multi-State Interaction: Handles complex UI states including filters, search bars, and expanding sections before data extraction.
- Resource Management: Automatically closes browser tabs and manages memory by preferring fewer, larger data writes over frequent tiny updates.
- Workspace Integration: Persistent storage of large datasets with automatic checkpointing for long-running scraping tasks.
SKILL.md
Loading
Related Openclaw Skills
Video Dubbing
An AI-driven guidance skill that facilitates high-quality video and audio dubbing workflows using the VideoAny platform.
Lobster Keeper
An automated maintenance skill that performs daily status checks, self-reflection, and memory synchronization for AI agents.
Agent Notify
A cross-platform notification system providing audio alerts and visual taskbar flashes for AI coding agents during task completion or confirmation requests.
List Layout Design & Optimization
A specialized tool for designing, optimizing, and auditing vertical list layouts to maximize content scannability and reading efficiency.
Content Claw AI
An automated engine that transforms source materials into platform-ready social media content, diagrams, and AI-generated visuals.
Chinese Copywriting Guidelines Full
A comprehensive AI agent skill that applies professional Chinese copywriting standards and typography rules to technical documentation, UI text, and marketing copy.
Featured*
