A professional-grade browser automation tool for extracting structured data from any website into CSV, JSON, or Excel formats.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install yinan-web-scraper
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install yinan-web-scraper using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Web Scraper?
The Web Scraper is a robust data extraction tool designed for Openclaw Skills users who need to transform unstructured web content into clean, actionable datasets. By leveraging agent-browser technology, it handles modern web complexities like JavaScript-rendered content, infinite scrolling, and multi-page navigation.
Whether you are monitoring competitors, building a research database, or tracking e-commerce trends, this skill provides the automation necessary to scale your data collection efforts efficiently. It bridges the gap between raw HTML and structured intelligence by providing a programmable interface for web interaction.
Web Scraper Use Cases
- Monitoring e-commerce product listings, prices, reviews, and inventory levels.
- Aggregating real estate property data, pricing trends, and agent contact information.
- Collecting job postings, salary requirements, and hiring trends from career boards.
- Scraping news articles, headlines, and publication dates for media and sentiment analysis.
- Building business directories and lead lists from web-based business listings.
How Web Scraper Works
- Identify the target URL and the specific data points you wish to extract using standard CSS selectors.
- Select the specialized script for the task, such as single page, paginated, or infinite scroll handlers.
- The skill utilizes browser automation to navigate the site, executing JavaScript and waiting for dynamic content to load.
- Fields are parsed and mapped according to your definitions, ensuring data consistency across different pages.
- The structured data is exported to your chosen format, including CSV, JSON, or Excel, ready for analysis.
Web Scraper Setup
To begin using these Openclaw Skills, ensure your environment is ready and execute the scripts via the command line:
# Example: Scrape a single page for product data
python scripts/scrape_page.py \
--url "https://example.com/products" \
--fields "title=h2.title,price=.price,link=a.href" \
--output products.csv
For paginated sites, use the following pattern:
python scripts/scrape_paginated.py \
--url "https://example.com/products?page={page}" \
--pages 10 \
--fields "title,price,description" \
--output all_products.csv
Web Scraper Data Schema & Taxonomy
The Web Scraper organizes data based on user-defined field names and CSS selectors. It supports several export schemas to fit your data pipeline:
| Format | Description |
|---|---|
| CSV | Flat file structure ideal for spreadsheet software and simple data imports. |
| JSON | Hierarchical structure including fields like scraped_at and nested object data. |
| XLSX | Multi-sheet Excel files with auto-fitted columns and professional formatting. |
Web Scraper Advanced Features
- Support for dynamic interactions via JSON-based action sequences (click, fill, scroll, and key press).
- Advanced handling of JavaScript-heavy sites using integrated agent-browser automation.
- Flexible pagination logic supporting both URL patterns and CSS-based next page button selectors.
- Intelligent rate limiting and delay configuration to manage anti-scraping measures within Openclaw Skills.
- Infinite scroll support for modern social media and feed-based web applications.
SKILL.md
Loading
Related Openclaw Skills
Task Automator
A comprehensive automation engine for Openclaw Skills designed to handle file management, data processing, and multi-service API integrations.
Social Content Generator
An AI-driven automation skill for generating platform-optimized social media posts, captions, and visual assets in bulk.
Price Monitor
Automate e-commerce price tracking and stock alerts using intelligent browser automation.
Inventory Manager
A professional-grade inventory management toolkit for syncing stock levels across Taobao, Douyin, and other major e-commerce platforms.
Planning with Files
A persistent markdown-based planning framework that provides AI agents with long-term memory for complex, multi-step tasks.
Water Reminder
A health-focused AI agent skill designed to manage hydration tracking and provide timely water consumption reminders.
Featured*
