A high-performance web crawler utilizing Cloudflare's Browser Rendering API to extract AI-ready Markdown and structured JSON from entire websites.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install cloudflare-crawl-skill
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install cloudflare-crawl-skill using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Cloudflare Crawl?
Cloudflare Crawl is an advanced tool designed to navigate the complexities of modern web scraping by leveraging Cloudflare's native infrastructure. Unlike traditional scrapers that struggle with bot detection, this Openclaw Skills integration runs directly within the Cloudflare ecosystem, ensuring seamless access to Cloudflare-protected sites. It provides a robust, job-based architecture for bulk content extraction.
The skill excels at transforming raw web pages into clean, AI-friendly formats. By utilizing the Browser Rendering API, it can execute JavaScript to capture dynamic content, making it an essential tool for developers building large-scale knowledge bases, RAG (Retrieval-Augmented Generation) pipelines, or comprehensive research datasets.
Cloudflare Crawl Use Cases
- Scraping entire documentation sites to build comprehensive RAG datasets for LLMs.
- Extracting structured product data from e-commerce sites using AI-powered JSON extraction.
- Researching competitors across multiple pages without being blocked by Cloudflare security headers.
- Automating the creation of searchable knowledge bases from legacy websites.
- Monitoring site changes by crawling sitemaps at scale using Openclaw Skills.
How Cloudflare Crawl Works
- The user initiates a crawl job by specifying a target URL, page limit, and desired output formats.
- A request is sent to the Cloudflare Browser Rendering API to start an asynchronous crawl task.
- The crawler discovers URLs via links or sitemaps, rendering JavaScript to ensure full content visibility.
- The skill polls the job status until completion, monitoring for running, completed, or errored states.
- Once finished, the extracted data is retrieved in Markdown, HTML, or structured JSON format for immediate use.
Cloudflare Crawl Setup
To use this skill within the Openclaw Skills ecosystem, you must configure your Cloudflare environment.
- Generate a Cloudflare API Token with Account.Browser Rendering permissions.
- Identify your Account ID from your Cloudflare dashboard.
- Export the required environment variables:
export CLOUDFLARE_API_TOKEN="your_token"
export CLOUDFLARE_ACCOUNT_ID="your_account_id"
- Start your first crawl job:
node scripts/crawl.js start https://example.com --limit 50
Cloudflare Crawl Data Schema & Taxonomy
The skill returns data in a structured format optimized for downstream processing and AI ingestion.
| Field | Description | Format |
|---|---|---|
| url | The specific page URL crawled | String |
| status | Completion status of the specific page | String |
| markdown | The main content of the page converted to Markdown | Markdown String |
| metadata | Includes page title and HTTP status codes | JSON Object |
| json | AI-extracted data if jsonOptions were provided | JSON Object |
Cloudflare Crawl Advanced Features
- AI-Powered Extraction: Use Workers AI to extract specific data points into structured JSON using natural language prompts.
- Sitemap Integration: Efficiently discover content by prioritizing XML sitemaps over recursive link crawling.
- Multi-format Support: Simultaneously generate HTML, Markdown, and JSON outputs in a single pass.
- Massive Scalability: Support for crawling up to 100,000 pages per job with a 7-day runtime limit.
- Native Bypass: Seamlessly access sites protected by Cloudflare WAF since the requests originate from within the Cloudflare network.
SKILL.md
Loading
Related Openclaw Skills
Progressive Insurance Skill
A comprehensive informational skill for navigating Progressive’s wide range of auto, home, and commercial insurance products and savings programs.
Creator Attribution Lite
A lightweight attribution model designed to connect content outputs directly to business outcomes and ROI.
Agent QA Gates
A multi-tiered validation framework designed to prevent hallucinations, data leaks, and formatting errors in AI agent outputs.
Agent Learner
A comprehensive AI toolkit for benchmarking, comparing, and optimizing agent prompts and evaluation results through persistent, file-based logging.
LaTeX Compiler
A high-performance LaTeX compilation engine for AI agents to generate professional PDFs from source strings.
CSV Wizard
An interactive command-line interface designed for professional data cleaning, featuring automatic type inference and flexible missing value strategies.
Featured*
