A production-grade, high-performance Node.js web crawler library optimized for large-scale crawling, concurrent execution, and robust proxy routing.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install node-crawler
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install node-crawler using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is node-crawler?
The node-crawler skill provides a robust, production-ready framework for managing large-scale web crawling and data harvesting workflows. Built specifically for high-throughput scenarios, this library manages complex operations such as server-side Cheerio parsing, custom connection pooling, and multi-threaded rate limiting, bypassing the need for complex, heavy browser automation tooling when structured HTTP scraping is sufficient. By integrating this capability with Openclaw Skills, developers can deploy fast, reliable web scraping tasks that can scale from dozens to millions of pages.
This library operates on top of got, featuring native support for HTTP/2, automatic character encoding detection, and intelligent retry structures. Unlike basic fetch mechanisms, it coordinates requests through an internal queue system to prevent target server overloads while maximizing performance.
node-crawler Use Cases
- Bulk Data Harvesting: Scraping dozens to millions of structured web pages efficiently for training data or catalog indexes.
- Batch Asset Downloading: Automating the download of hundreds of binary files (PDFs, images, zip files) with built-in retry logic and session integrity.
- Long-Running Web Spiders: Running persistent scraping processes that require dynamic rate limits, connection limits, and auto-retries.
- Deep Link Pagination: Crawling nested web structures, automatically following pagination links, and managing cascading discovery flows.
- Proxy-Protected Crawling: Accessing content across restricted domains using dynamic proxy rotation and isolated per-proxy rate limiters.
How node-crawler Works
- Initialization: Configure the crawler instance with rules for concurrency (maxConnections), rate limiting, retries, and default parsing parameters.
- Enqueueing Tasks: Add target URLs to the execution queue using crawler.add(). Custom parameters can be attached to each request using userParams for downstream contextual execution.
- Request Scheduling: The internal scheduler throttles requests based on per-domain rate limits, assigns custom proxies dynamically, and initiates raw HTTP/2 or HTTP/1 requests.
- HTML Parsing & Response: Upon a successful HTTP handshake, the response body is processed (and parsed via Cheerio by default, exposing a jQuery-like selection interface).
- Callback Execution: The specified callback handler extracts structured elements or writes files. Calling the done() function is mandatory to free up connection slots.
- Drain State: Once the queue is fully depleted, the system triggers the 'drain' event, signaling workflow completion.
node-crawler Setup
To utilize this capability within Openclaw Skills, you must ensure your development environment runs Node.js 22 or higher. The package is built as pure ESM.
First, install the package using your preferred manager:
npm install crawler
If your host application strictly relies on CommonJS, ensure you target the beta release:
npm install crawler@beta
Configure your entry point file to import the module:
import Crawler from "crawler";
node-crawler Data Schema & Taxonomy
The node-crawler skill standardizes execution state and request-response metadata. The following table describes how core request configuration, response payloads, and context parameters are modeled:
| Object Property | Type | Description |
|---|---|---|
res.body |
String \ | Buffer |
res.$ |
Function | The pre-loaded Cheerio (jQuery-style) parser, initialized when jQuery: true. |
res.options.userParams |
Object | Arbitrary metadata or state variables passed down through the queue lifecycle. |
options.proxy |
String | Active proxy URL assigned dynamically during the scheduling phase. |
options.rateLimiterId |
Number \ | String |
node-crawler Advanced Features
- Dynamic Proxy Rotation: Register a listener on the 'schedule' event to inject rotating proxy credentials programmatically on a per-request basis.
- Parallel Rate Limiting: Isolate scraping behaviors across different domains by declaring explicit rateLimiterId boundaries to prevent domain-level blacklisting.
- Binary Stream Pipeline: Download rich media securely without memory bloat by disabling Cheerio parsing and capturing the raw stream buffer directly.
- HTTP/2 Multiplexing: Maximize connection efficiency over modern networks by enabling the native http2 configuration flag, bypassing heavy browser-based DOM overhead in your Openclaw Skills architecture.
- Lifecycle Middleware Hooking: Execute custom code during 'schedule', 'request', and 'drain' events to build self-healing crawling workflows.
SKILL.md
Loading
Related Openclaw Skills
The Red Tent
An immersive AI companion skill based on Anita Diamant's novel, empowering users to reclaim silenced narratives, foster female community, and explore the sacred traditions of womanhood.
Eats, Shoots & Leaves
An AI-powered grammar and punctuation assistant inspired by Lynne Truss's bestselling book, designed to help you write with absolute clarity and precision.
Perception Music
An offline, zero-dependency FM synthesizer built in Zig that translates your AI agent's real-time perception data into unique electronic music.
Job Search — Serbia & Remote (job-search-belgrade)
An intelligent AI coding agent skill that scrapes regional and remote job boards, cross-references findings against your PDF resume, and outputs sorted listings with fit scores directly in your terminal or via SMTP email digests.
Sentiment Cycle Analyzer
A quantitative market sentiment analyzer for A-share trading that computes a 7-dimensional score to pinpoint market phases and detect inflection points.
copy-brain
A specialized agent skill to clone and run the decision-making framework, mental models, and analytical style of any public figure.
Featured*
