Firecrawl is a high-performance web scraping and crawling engine designed to transform entire websites into clean, structured markdown for AI applications.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install web-scraper-firecrawl
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install web-scraper-firecrawl using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Firecrawl?
Firecrawl is a specialized tool within the Openclaw Skills library that simplifies the complex process of turning web content into LLM-ready data. It bypasses traditional scraping hurdles by providing a clean API for single-page extraction, site-wide crawling, and URL mapping.
By focusing on markdown output, it ensures that gathered information is perfectly formatted for RAG (Retrieval-Augmented Generation) systems, knowledge bases, and AI agent training. Whether you are dealing with JavaScript-heavy single-page applications or massive documentation sites, this skill provides the reliability and structure needed for professional-grade data harvesting.
Firecrawl Use Cases
- Automating the creation of local knowledge bases from online documentation for RAG systems.
- Extracting structured product information, pricing, and metadata from e-commerce sites.
- Discovering and mapping entire site architectures to identify specific content sub-directories.
- Migrating legacy CMS content into clean markdown for modern static site generators.
- Monitoring competitor blogs or news sites for specific updates using date-based mapping.
How Firecrawl Works
- The user initiates a request via the CLI, specifying a target URL and the desired action (scrape, crawl, map, or extract).
- The skill authenticates with the Firecrawl API using a pre-configured environment variable.
- For scraping, the engine fetches the page, handles JavaScript rendering if necessary, and strips away boilerplate content to isolate the main text.
- For crawling, the engine recursively follows links based on defined depth limits and inclusion/exclusion patterns.
- The content is processed into the requested format—typically clean markdown or structured JSON—and saved to the specified output directory.
- Integrated retry logic and rate limiting ensure the process completes even for large-scale data extractions.
Firecrawl Setup
To get started with Firecrawl in your collection of Openclaw Skills, follow these steps:
- Obtain a Firecrawl API key from firecrawl.dev.
- Install the necessary Python dependencies:
pip install requests
- Configure your environment variable:
export FIRECRAWL_API_KEY="fc-your-api-key"
Firecrawl Data Schema & Taxonomy
The Firecrawl skill organizes data based on the command executed. Below is the typical output structure:
| Data Type | Format | Metadata Included |
|---|---|---|
| Scrape | Markdown, HTML | Page title, source URL, alt text for images |
| Crawl | Directory of Files | File per URL, directory hierarchy preservation |
| Map | List of Strings | Array of discovered URLs within the domain |
| Extract | JSON | Custom keys defined by user-provided CSS selectors |
Firecrawl Advanced Features
- Support for JavaScript rendering using the --wait-for flag to ensure content from SPAs is captured.
- Advanced regex-based include and exclude patterns to precisely target specific subfolders during site-wide crawls.
- Structured data extraction using custom CSS schemas to turn unstructured web pages into valid JSON objects.
- Batch processing capabilities to handle hundreds of URLs simultaneously with configurable concurrency limits.
- Seamless integration with other Openclaw Skills like markdown-sync-pro for automated content distribution to GitHub or Notion.
SKILL.md
Loading
Related Openclaw Skills
Seven: The Cultural & Scientific Intelligence of 7
A comprehensive bilingual knowledge skill providing deep cultural, scientific, and symbolic insights into the number 7, alongside the OpenClaw project matrix.
Cultural Eight: Cross-Cultural Digital Insights
A dual-language knowledge skill providing deep cultural, philosophical, and modern insights into the significance of the number 8.
vefaas
A powerful CLI tool for deploying and managing serverless functions and web applications on Volcengine veFaaS.
Imprint
Imprint is an adaptive operator modeling skill that enables AI agents to learn user preferences and decision patterns through passive observation rather than manual configuration.
Sector Charts
A visualization tool designed to map competitive positioning using multi-dimensional bubble charts for market analysis.
osv-ui CVE Audit Skill
A comprehensive security auditing skill that scans for CVE vulnerabilities and provides a visual dashboard for managed dependency remediation.
Featured*
