An instruction-based skill that teaches AI agents how to efficiently drive the Firecrawl API to scrape, crawl, search, and extract clean, cited Markdown content from the web.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install firecrawl-api-al
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install firecrawl-api-al using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Firecrawl Web Scraping Skill?
The Firecrawl Web Scraping Skill provides instructional knowledge that teaches an AI agent exactly when and how to leverage Firecrawl to perform clean, safe, and cost-effective web extraction. Rather than functioning as executable infrastructure, this skill acts as a cognitive playbook for AI coding agents to interact with Firecrawl MCP servers or HTTP endpoints. It empowers agents to handle complex client-side JavaScript rendering, bypass anti-bot measures, and transform noisy HTML into clean, LLM-ready Markdown.
By integrating this skill into Openclaw Skills, developers enable their AI agents to make smart structural decisions: executing target-rich scraping, running asynchronous multi-page crawls, and performing schema-backed structured data extractions. It ensures optimal credit utilization and establishes a robust citation provenance mechanism, preventing security exploits like prompt injections from untrusted web text.
Firecrawl Web Scraping Skill Use Cases
- Retrieving Clean Web Content for RAG: Convert JavaScript-heavy target URLs into clean Markdown, stripping headers, footers, ads, and boilerplate elements automatically.
- Bulk Section Crawling: Map and crawl nested documentation or blog directories asynchronously, applying strict URL boundaries and limits.
- Schema-Based Structured Extraction: Leverage Firecrawl's scraping capabilities to turn raw pages into highly structured, validated JSON data matching a specific JSON schema.
- Web Search & Synthesis: Conduct cost-efficient web searches and selectively scrape top-ranking results to construct authoritative answers.
- URL Discovery & Pre-mapping: List site-wide URLs cleanly before committing expensive compute resources to crawling or scraping workflows.
How Firecrawl Web Scraping Skill Works
- Trigger Assessment: The AI agent analyzes the user prompt to decide whether Firecrawl is required or if local context can satisfy the request.
- Path Selection: The agent selects the optimal operation (
scrape,crawl,map, orsearch) based on whether it needs a single page, dynamic search query results, URL discovery, or deep-domain crawling. - Request Optimization: Parameter payloads are configured to request minimal formats (defaulting to
markdownor target structured JSON), set limits, apply path filters, and invoke actions likewaitForfor dynamic layouts. - Async Execution & Polling: For asynchronous crawl actions, the agent initiates the job, retrieves the token ID, and polls the endpoint with exponential backoff until the job reports completion.
- Post-Processing & Safety Checks: The agent extracts output, parses metadata for citations (
sourceURL), checks credit logs, validates schemas, and treats scraped pages as untrusted data, filtering out potential prompt injection attacks.
Firecrawl Web Scraping Skill Setup
To configure this skill with your agent within the Openclaw Skills ecosystem, ensure the required credentials are set up. This skill operates as instructional knowledge directing your tools or your direct HTTP connections.
1. Set the API Key
Ensure the FIRECRAWL_API_KEY is present in your environment variables. Never hardcode or print this key in logs:
export FIRECRAWL_API_KEY="your_firecrawl_api_key_here"
2. Configure MCP Server (Optional)
If you are using the Model Context Protocol (MCP), configure your host settings (e.g., claude_desktop_config.json) to invoke the Firecrawl MCP server:
{
"mcpServers": {
"firecrawl": {
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-firecrawl"],
"env": {
"FIRECRAWL_API_KEY": "your_firecrawl_api_key_here"
}
}
}
}
Ensure that tool calls mapping to firecrawl_scrape, firecrawl_crawl, firecrawl_map, and firecrawl_search are fully exposed to the agent.
Firecrawl Web Scraping Skill Data Schema & Taxonomy
Firecrawl operations standardly organize data into structured JSON objects. The agent maintains a strict hierarchy to isolate data, status responses, and usage metrics:
Scrape & Extract Schema
| Field Name | Type | Description |
|---|---|---|
success |
Boolean | Confirms whether the HTTP request or tool invocation succeeded. |
data |
Object | Root container for the scraped output components. |
data.markdown |
String | Clean, LLM-ready markdown conversion of the main web page content. |
data.html |
String | Raw extracted HTML (only requested if raw selectors/structures are required). |
data.json |
Object | The structured JSON data extracted dynamically when utilizing schema validation. |
data.metadata |
Object | Contains critical session information such as statusCode, sourceURL, and page metrics. |
creditsUsed |
Integer | Tracks usage billing per execution loop to enforce cost-control boundaries. |
Asynchronous Crawl Job Schema
| Endpoint/State | Response Payload Structure | Purpose |
|---|---|---|
POST /v2/crawl |
{ "success": true, "id": "crawl-job-id" } |
Initiates the asynchronous crawling runner. |
GET /v2/crawl/{id} |
{ "status": "scraping"|"completed", "completed": 15, "total": 50, "data": [...], "next": "url" } |
Tracks progress status, allows pagination processing via next, and aggregates final pages. |
Firecrawl Web Scraping Skill Advanced Features
- Interactive Browser Actions: Go beyond static scraping by executing pre-render workflows like clicking, scrolling, typing, or navigating deep frames dynamically using the
actionsschema block. - Intelligent Path Exclusions: Minimize crawler drift and limit budget exposure by utilizing complex regex patterns inside
includePathsandexcludePathsfilters. - Resilient Polling with Stateful Recovery: Preserves agent state on crawl job timeouts, recording raw transaction
ids to resume collection instead of triggering redundant cycles. - Cached Retrieval Engine: Leverages the
maxAgeparameter to optimize retrieval speeds and balance fresh web content requirements against credit spend budgets. - Self-Correcting Rate Mitigation: Leverages precise diagnostic classification of HTTP codes (401, 400, 402, 429, 5xx) to perform target-specific recovery actions like exponential backoff and rate-limit throttling.
SKILL.md
Loading
Related Openclaw Skills
Tavily Web Search Skill
The Tavily Web Search Skill provides comprehensive instructional knowledge that teaches AI coding agents exactly when and how to leverage Tavily for live web searches, deep site crawls, content extraction, and mapping.
OpenAI Agent Skill
An operations guide and toolset for AI agents integrating OpenAI MCP services to generate text, compute embeddings, run reasoning tasks, produce images, and transcribe audio.
lyra-prompt-architect
An advanced prompt engineering skill that builds precise, high-performance prompts from scratch through structured dialogue and advanced reasoning frameworks.
D3.js Visualization
A powerful, low-level JavaScript library for binding arbitrary data to a Document Object Model and applying data-driven transformations.
Cross-Platform Notifier
A unified system notification utility that enables AI agents to send native desktop alerts across Windows, macOS, Linux, and WSL.
Exa Web Search Skill
A powerful neural and keyword web search skill that equips AI agents with real-time web access, clean content extraction, similarity mapping, and citation-grounded answering via Exa.
Featured*
