An enterprise-grade data collection engine designed to bypass anti-bot measures while providing structured data extraction.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install smart-crawler
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install smart-crawler using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Smart Crawler?
Smart Crawler is a robust enterprise solution for high-volume web data collection and sophisticated anti-detection management. Built as part of the Openclaw Skills library, it leverages powerful rendering engines like Playwright and Selenium to navigate dynamic, JavaScript-heavy websites that traditional scrapers fail to access.
By integrating advanced features such as automatic User-Agent rotation, proxy pool management, and request frequency throttling, Smart Crawler ensures high success rates and reliability. It is designed for developers who need a scalable, distributed system capable of handling complex scraping tasks without being blocked by modern security protocols.
Smart Crawler Use Cases
- Scaling data collection across multiple nodes using Redis-backed distributed queues.
- Scraping dynamic web applications (SPAs) that require full JavaScript rendering.
- Bypassing sophisticated anti-bot systems through automated behavior simulation and proxy rotation.
- Extracting structured data from unstructured HTML using CSS selectors, XPath, or Regex.
- Automating data cleaning and standardization workflows to ensure high-quality datasets.
How Smart Crawler Works
- Initialize the CrawlerEngine with preferred rendering drivers (Playwright or Selenium) and proxy settings.
- Apply anti-detection configurations including custom User-Agent strings and rate limiting parameters.
- Dispatch crawl requests to target URLs with defined extraction rules (XPath, CSS, or Regex).
- Execute dynamic rendering to capture content from AJAX-heavy or single-page applications.
- Process the raw results through a data cleaning layer to remove duplicates and normalize formatting.
- (Optional) Synchronize tasks through Redis for distributed execution across a cluster.
Smart Crawler Setup
To integrate this tool into your Openclaw Skills workflow, follow these steps:
# Clone the repository and install dependencies
pip install -r requirements.txt
# Install the necessary browser binaries for Playwright
playwright install
# Verify the installation by running the test suite
cd tests
python test_crawler.py
Smart Crawler Data Schema & Taxonomy
Smart Crawler organizes its core modules and data extraction logic as follows:
| Module | Purpose | Key Metadata/Outputs |
|---|---|---|
crawler_engine.py |
Core Execution | Status codes, page load times, raw HTML content |
proxy_manager.py |
Network Layer | Proxy IP, latency, rotation logs |
data_extractor.py |
Parsing Layer | JSON/Dictionary of structured fields (Title, Content, etc.) |
anti_detection.py |
Security | Fingerprint profiles, User-Agent strings |
Smart Crawler Advanced Features
- Redis-backed distributed scraping for high-concurrency enterprise workloads.
- Automated proxy pool management with health checks and latency monitoring.
- Built-in data cleaning pipeline for de-duplication and sensitive information filtering.
- Seamless integration with the broader Openclaw Skills ecosystem for multi-agent automation.
- Headless and headful mode switching for easier debugging of complex navigation flows.
SKILL.md
Loading
Related Openclaw Skills
iFly PDF & Image OCR
A powerful OCR skill that leverages iFlytek's advanced LLM-based recognition to extract text and structure from images and PDF documents.
MediaIO Seedream Image Generator
A professional-grade image generation skill leveraging ByteDance Seedream 4.0 for high-aesthetic 4K visuals with style and character consistency.
prose-optimize
A specialized optimization skill designed to minimize LLM API calls and token consumption within .prose workflow files.
Global Insights
A comprehensive intelligence skill providing real-time data on cross-border e-commerce, international finance, and global lifestyle services.
Exa Web Search (Free)
A neural search skill for Openclaw Skills that enables AI agents to perform web, code, and company research without requiring an API key.
TimeFriend
A natural language-driven productivity skill for logging time, capturing reflections, and managing todo lists through simple conversation.
Featured*
