A specialized tool that bypasses web scrapers blocks by using alternative proxy services to fetch webpage content in Markdown format.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install web-content-fetcher
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install web-content-fetcher using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Web Content Fetcher?
The Web Content Fetcher is a robust utility designed for AI agents that need to access web data when standard crawling methods are restricted or filtered. By utilizing high-reliability services like Jina AI, Markdown.new, and Defuddle, this skill ensures that your agent can always retrieve the information it needs in a clean, LLM-friendly Markdown format. As a core component of the Openclaw Skills library, it provides a seamless fallback mechanism for handling sites protected by Cloudflare or complex anti-bot measures.
Web Content Fetcher Use Cases
- Extracting technical documentation from sites that block standard curl requests.
- Converting news articles and blog posts into Markdown for summary generation.
- Bypassing Cloudflare wait screens or bot detection on research targets.
- Providing a reliable fallback when default web search tools return empty or filtered results.
How Web Content Fetcher Works
- The AI agent attempts to access a URL using standard fetch tools.
- If the initial attempt fails or is blocked, the agent triggers the Web Content Fetcher skill.
- The tool identifies the optimal service provider (r.jina.ai, markdown.new, or defuddle.md) based on the target site's protection level.
- A request is routed through the selected service to scrape and convert the HTML content.
- The resulting Markdown is returned to the agent's context for immediate processing.
Web Content Fetcher Setup
To use this tool within your agent environment, you can call the provided shell script directly. Ensure you have network access to the proxy services used by these Openclaw Skills.
# Fetch a webpage using the default preferred service (Jina)
./fetch.sh https://example.com
# Specify a specific bypass method for Cloudflare sites
./fetch.sh https://example.com markdown
Web Content Fetcher Data Schema & Taxonomy
The skill operates on a simple input-output model to maintain compatibility with various AI workflows.
| Parameter | Type | Description |
|---|---|---|
| url | string | The full URL of the webpage to be converted |
| method | string | Optional: select between jina (default), markdown, or defuddle |
| output | string | The fetched content returned as a Markdown-formatted string |
Web Content Fetcher Advanced Features
- Multi-service redundancy: Automatically switches between providers if the primary fetch fails.
- Cloudflare optimization: Includes specialized handling for sites using Cloudflare protection via the markdown.new endpoint.
- CLI Integration: Can be used as a standalone command-line tool or integrated into complex Openclaw Skills pipelines.
- Low-latency response: Uses optimized proxy endpoints to ensure fast content delivery for real-time agent interactions.
SKILL.md
Loading
Related Openclaw Skills
Onnex YouTube
A security-reviewed tool for fetching YouTube transcripts, 4K videos, and channel metadata without API quotas.
Clawchemy
Clawchemy is a decentralized element discovery game where AI agents use LLMs to create new elements that are automatically minted as tokens on the Base chain.
Torch Liquidation Agent
A technical scanner and automation bot for monitoring loan health and executing liquidations on the Solana-based Torch Market protocol.
Torch Domain Auction Bot
A sophisticated Solana bot that tokenizes domains and manages automated lending, monitoring, and liquidation cycles as part of the Openclaw Skills ecosystem.
IDA – Instructional Design Agent
A strategic learning engine that transforms raw discovery data into structured instructional blueprints and performance recommendations.
RepoMedic
RepoMedic is an AI agent skill designed to safely triage and remediate GitHub dependency issues, security vulnerabilities, and lockfile inconsistencies with strict safety guardrails.
Featured*
