A production-grade web scraping and browser automation library for JavaScript and Python designed to handle anti-blocking and high-concurrency crawling.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install crawlee
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install crawlee using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Crawlee?
Crawlee is a comprehensive toolkit for developers building reliable web scrapers and crawlers. As a premier integration within the Openclaw Skills ecosystem, it provides the necessary abstractions to manage complex scraping tasks such as session persistence, proxy rotation, and browser fingerprinting. Whether you are using Node.js or Python, it enables your AI agents to interact with the web like a human user.
By unifying different crawling technologies—ranging from fast HTML parsers like Cheerio and BeautifulSoup to full-head-less browser automation via Playwright and Puppeteer—this library ensures that data extraction is both efficient and difficult to detect. Utilizing this skill within Openclaw Skills allows for the creation of robust data pipelines that can scale across thousands of pages while maintaining high performance.
Crawlee Use Cases
- Extracting structured product information, prices, and reviews from e-commerce platforms.
- Automating web-based workflows and interactions for testing or data entry tasks.
- Performing large-scale web crawls while bypassing sophisticated anti-bot and rate-limiting systems.
- Gathering competitive intelligence by monitoring website changes and news feeds.
- Capturing automated screenshots or downloading PDF documents from web pages at scale.
How Crawlee Works
- Initialize the crawler by defining a list of starting URLs or a RequestQueue to manage dynamic discovery.
- Choose the appropriate engine (Cheerio, BeautifulSoup, Playwright, or Puppeteer) based on whether the target site requires JavaScript rendering.
- Implement a request handler to define data extraction logic, such as selecting HTML elements or enqueuing found links into the queue.
- The system automatically handles retries, proxy rotation, and session management to ensure a high success rate even on protected sites.
- Extracted results are stored in the Dataset, while assets like images or state metadata are saved in the KeyValueStore for persistence.
Crawlee Setup
For JavaScript or TypeScript projects, use the following commands to get started with the Openclaw Skills environment:
npx crawlee create my-crawler
cd my-crawler && npm install
# For browser automation support
npm install crawlee playwright && npx playwright install
For Python projects, install the library and its optional dependencies via pip:
pip install crawlee
# For BeautifulSoup support
pip install 'crawlee[beautifulsoup]'
# For Playwright support
pip install 'crawlee[playwright]' && playwright install
Crawlee Data Schema & Taxonomy
The skill manages data across several storage types, ensuring state is preserved between runs and results are easy to export. By default, these are stored in the ./storage directory.
| Storage Component | Directory | Function |
|---|---|---|
| Dataset | ./storage/datasets/default/ |
Stores structured results as individual JSON objects (one per file). |
| KeyValueStore | ./storage/key_value_stores/default/ |
Holds binary files, screenshots, and global application state. |
| RequestQueue | ./storage/request_queues/default/ |
Manages the list of URLs to be crawled and tracks their status (pending/handled). |
Crawlee Advanced Features
- Automatic Fingerprint Rotation: Mimics real browser fingerprints to bypass bot detection without manual configuration.
- Tiered Proxy Management: Smart escalation between different proxy levels (datacenter to residential) based on target response.
- Adaptive Crawling: Automatically determines if a page requires a full browser or can be processed with faster HTTP parsers.
- Intelligent Concurrency: Dynamically adjusts the number of parallel tasks based on available CPU and memory resources.
- Integrated SessionPool: Maintains consistent user identities, cookies, and headers across multiple requests to simulate real user sessions.
SKILL.md
Loading
Related Openclaw Skills
Bioinfo Daily
An automated daily report generator that tracks high-impact bioinformatics and oncology research using the PubMed API.
SkillBoss
A unified multi-AI gateway providing access to over 50 models for chat, image generation, video, and speech through a single API key.
Alibaba Cloud CosyVoice Voice Design
Create custom AI voices using natural language descriptions with Alibaba Cloud CosyVoice models.
Gold Analysis
A comprehensive gold market analysis tool providing real-time pricing, technical indicators, and macroeconomic fundamental research.
Alibaba Cloud CosyVoice Voice Clone
A professional AI integration for enrolling and cloning custom voices using Alibaba Cloud Model Studio CosyVoice models.
Alibaba Cloud Realtime ASR (Qwen)
A low-latency realtime speech recognition skill leveraging Alibaba Cloud Model Studio Qwen ASR models for streaming audio processing.
Featured*
