An intelligent web data collection tool that automatically identifies page structures to extract lists, tables, and details into structured formats.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install xh-smart-scraper
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install xh-smart-scraper using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Smart Web Scraper?
Smart Web Scraper is a powerful data extraction engine designed for Openclaw Skills users who need to transform unstructured web content into actionable data. It leverages intelligent recognition to automatically detect the layout of list pages and detail views, minimizing the need for manual selector configuration. Whether you are gathering e-commerce pricing, real estate listings, or job postings, this skill provides a robust framework for high-volume data harvesting.
The tool is built with a focus on reliability and ethical scraping. It features built-in anti-crawling defenses to handle modern website protections, ensuring that your data collection workflows remain uninterrupted. By integrating this into your Openclaw Skills ecosystem, you gain the ability to export web data directly into JSON, CSV, or Excel formats, or even sync it to live databases.
Smart Web Scraper Use Cases
- E-commerce Product Monitoring: Automatically gather product names, prices, and images from online marketplaces.
- Market Research: Scrape real estate listings or financial data for competitive analysis.
- Lead Generation: Extract job titles and company information from professional recruitment portals.
- Content Aggregation: Batch collect articles, titles, and metadata for news or blog archives.
How Smart Web Scraper Works
- Target Identification: The skill receives a URL or configuration file defining the target website and desired fields.
- Page Parsing: It utilizes intelligent recognition to scan the DOM for repeated patterns, identifying lists, tables, and pagination elements.
- Anti-Bot Navigation: The engine applies randomized User-Agents and request delays to mimic human behavior and avoid IP blocking.
- Data Extraction: Key fields such as text, attributes, and images are scraped according to the defined selectors or automated discovery logic.
- Structured Export: The collected data is cleaned and saved into the user's preferred format (JSON, CSV, or Excel) or pushed to a database.
Smart Web Scraper Setup
To get started with this component of your Openclaw Skills toolkit, follow these installation steps:
# Navigate to the scraper directory
cd smart-web-scraper
# Install required dependencies
npm install
Smart Web Scraper Data Schema & Taxonomy
Smart Web Scraper organizes data logically based on the fields identified during the extraction process. Users can define custom taxonomies using a config.json file:
| Feature | Description |
|---|---|
| target | Defines the URL, total pages to scrape, and wait conditions. |
| fields | Array of objects specifying the name, CSS selector, and data type (text/attr). |
| export | Configures the output format (JSON/CSV/Excel) and destination file path. |
| anti-bot | Manages User-Agent rotation and randomized delay intervals. |
Smart Web Scraper Advanced Features
- Multi-page Batch Collection: Supports automatic pagination to crawl through deep web directories.
- Database Integration: Direct support for saving scraped results to MySQL or MongoDB.
- Intelligent Field Mapping: Automatically extracts common keywords like title, price, and author.
- Customizable Anti-Crawling Strategy: Includes IP proxy pool support and automatic retry mechanisms for failed requests.
- Flexible CLI Interface: Run quick scrapes or complex configurations directly from the terminal within the Openclaw Skills environment.
SKILL.md
Loading
Related Openclaw Skills
Weekend Flights Assistant
A specialized AI agent skill for discovering and booking optimized 2-3 day weekend escapes with real-time pricing via Fliggy.
Travel Weather & Season Forecast
An AI agent skill for real-time weather forecasts, seasonal travel planning, and integrated travel booking via the flyai CLI.
Travel Weather Skill
A professional CLI-integrated skill for AI agents to provide live weather data, seasonal travel recommendations, and real-time booking links.
Subtitle Assistant
An AI-powered utility for downloading, summarizing, and translating YouTube video subtitles directly from the terminal.
Auto Workflow Pro
An advanced automation engine designed to orchestrate complex task chains through cron scheduling, webhooks, and file monitoring.
Xueqiu Collector (xueqiu-collector)
A comprehensive automation skill for scraping, analyzing, and organizing full user post histories from the Xueqiu financial social network.
Featured*
