An advanced asynchronous web crawler utilizing Cloudflare Browser Rendering for high-speed data extraction and AI-powered structured processing.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install cf-crawl
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install cf-crawl using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Cloudflare Crawl (cf-crawl)?
The cf-crawl skill is a high-performance automation tool designed for developers needing to ingest entire websites or complex web structures into their AI agents. By leveraging the Cloudflare Browser Rendering /crawl API, it allows for seamless transition between fast, cost-effective HTML fetching and full headless browser rendering for JavaScript-heavy Single Page Applications (SPAs). This addition to the Openclaw Skills library enables agents to build comprehensive local knowledge bases from online documentation or live sites.
Beyond simple fetching, cf-crawl integrates directly with Workers AI for structured data extraction. Users can provide natural language prompts or JSON schemas to transform unstructured web content into machine-readable data at scale. With built-in support for job polling and R2 caching, it handles the complexities of long-running crawl operations without blocking agent workflows.
Cloudflare Crawl (cf-crawl) Use Cases
- Building local knowledge bases for RAG (Retrieval-Augmented Generation) by crawling documentation sites.
- Extracting structured product data, pricing, and descriptions from e-commerce platforms using AI-powered JSON prompts.
- Automating content audits by following link depths and filtering specific URL patterns.
- Capturing content from dynamic, JavaScript-rendered websites that standard fetch tools cannot process.
How Cloudflare Crawl (cf-crawl) Works
- The user initiates a crawl job via the crawl.sh script, specifying the target URL and parameters like depth, limit, and format.
- The skill sends an asynchronous request to the Cloudflare Browser Rendering API, which assigns a unique Job ID.
- Cloudflare's infrastructure handles the link following, optional JS rendering, and content extraction in the background.
- The poll.sh script monitors the Job ID status until the crawl is complete or the timeout is reached.
- Upon completion, the results are retrieved in the requested format (Markdown, HTML, or JSON) and saved locally for agent use.
Cloudflare Crawl (cf-crawl) Setup
First, ensure your Cloudflare credentials are configured in the secrets directory:
# Edit ~/.clawdbot/secrets/cloudflare-crawl.env
CF_ACCOUNT_ID=<your_account_id>
CF_CRAWL_API_TOKEN=<your_api_token>
To run a basic crawl immediately:
bash ~/clawd/skills/cf-crawl/scripts/crawl.sh "https://example.com" --limit 5 --format markdown
Cloudflare Crawl (cf-crawl) Data Schema & Taxonomy
The skill organizes output based on the job lifecycle and user-specified formats:
| Data Component | Description |
|---|---|
| Job Metadata | Includes Job ID, status, and page count logs. |
| Content Output | Saved as .json, .md, or .html files containing page body and metadata. |
| Structured Data | If using --format json, the output conforms to a provided --json-schema or AI-generated schema. |
| Caching | Results are cached in Cloudflare R2 with a default 24-hour TTL (max-age). |
Cloudflare Crawl (cf-crawl) Advanced Features
- AI-Powered Extraction: Use
--json-promptto let Workers AI identify and extract specific data points automatically. - Pattern Filtering: Advanced URL control using
--includeand--excludewildcard patterns to target specific site sections. - Discovery Methods: Toggle between sitemaps, link following, or both for optimized URL discovery.
- Headless Switching: Enable
--renderonly when necessary to manage costs while ensuring compatibility with modern web frameworks.
SKILL.md
Loading
Related Openclaw Skills
Shuidibao Assistant
An intelligent insurance consultant for querying, comparing, and receiving purchase advice for Shuidibao insurance products.
Gold Price Tracker
A specialized skill for retrieving real-time market prices for gold, silver, and other precious metals across banks and retail brands.
TimeBlock
TimeBlock is a local-first CLI utility for logging, tracking, and analyzing intentional time-blocking operations through simple bash commands.
Gold Price Summary
A comprehensive financial intelligence tool that provides real-time international and domestic gold price analysis and structured investment advice.
cside Site Scanner
A comprehensive security auditing tool that scans websites for third-party script risks, tracking methods, and compliance gaps.
Popup Generator
A professional framework for designing, triggering, and auditing high-conversion popups and modals without sacrificing user experience or SEO.
Featured*
