An AI-driven web scraping tool designed to automatically recognize and extract structured data like lists and tables from any URL.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install smart-scraper
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install smart-scraper using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Smart Scraper?
Smart Scraper is a sophisticated utility within the Openclaw Skills ecosystem designed to eliminate the manual effort of writing custom selectors for every website. By leveraging intelligent structure recognition, it identifies the underlying layout of a page to accurately parse content regardless of the DOM complexity.
Whether you are dealing with infinite-scrolling product feeds or deeply nested data tables, this skill provides a streamlined interface for developers to turn unstructured web pages into clean, actionable datasets. It bridges the gap between raw HTML and structured formats like JSON, CSV, and Markdown.
Smart Scraper Use Cases
- Extracting product listings and pricing data from e-commerce sites for market analysis.
- Converting long-form blog posts and news articles into Markdown for personal knowledge bases.
- Harvesting tabular data from financial or research portals into CSV format for spreadsheet processing.
- Monitoring news aggregators or forum lists for specific top-level trends and metadata.
How Smart Scraper Works
- The user provides a target URL and optional extraction parameters via the CLI.
- Smart Scraper initializes a headless browser session using Playwright to load the page content.
- The engine analyzes the page layout to detect patterns corresponding to lists, articles, or tables.
- If enabled, the tool performs auto-scrolling to capture lazy-loaded content or infinite feeds.
- The identified data is parsed for specific attributes like dates, URLs, and prices.
- The extracted data is serialized into the requested format (JSON, CSV, or Markdown) and output to the terminal or a file.
Smart Scraper Setup
To integrate Smart Scraper into your workflow, ensure you have Node.js and npm installed. You can set up the required dependencies for this entry in Openclaw Skills using the following commands:
npm install playwright
npx playwright install
Smart Scraper Data Schema & Taxonomy
Smart Scraper organizes output based on the extraction type and format selected. The internal taxonomy follows this structure:
| Data Type | Attributes Extracted | Recommended Format |
|---|---|---|
| List | Titles, Links, Prices, Image URLs | JSON / CSV |
| Article | Headlines, Body Text, Author, Date | Markdown |
| Table | Header rows, Columnar data, Cell values | CSV |
Metadata includes the source URL and the timestamp of the extraction session.
Smart Scraper Advanced Features
- Intelligent Auto-detection: Use the
--type autoflag to let the AI determine the best extraction strategy for the page structure. - Infinite Scroll Support: The
--scrollflag enables the agent to handle modern, reactive web apps that load content dynamically. - Flexible Serialization: Seamlessly switch between data structures for different downstream applications, such as using CSV for Excel or Markdown for Obsidian.
- Item Capping: Use the
--maxparameter to optimize performance and token usage by limiting the extraction count.
SKILL.md
Loading
Related Openclaw Skills
Amazon Leadership Principles for Product & Engineering
Integrate Amazon's legendary Leadership Principles into your AI-assisted coding and product development workflow.
TimeFriend
A natural language-driven productivity skill for logging time, capturing reflections, and managing todo lists through simple conversation.
Exa Web Search (Free)
A neural search skill for Openclaw Skills that enables AI agents to perform web, code, and company research without requiring an API key.
Smart Crawler
An enterprise-grade data collection engine designed to bypass anti-bot measures while providing structured data extraction.
Shadow Trading Dashboard
A professional-grade simulation environment for managing and visualizing A-share shadow trading portfolios and real-time performance metrics.
Baichuan AI Hot Trend
A specialized automation skill for organizing and summarizing public information from Baichuan AI's official documentation and product pages.
Featured*
