A powerful utility for AI agents to retrieve web content from known URLs and convert it into structured markdown, text, or image files.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install shikamaru-web-fetch
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install shikamaru-web-fetch using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Web Fetch?
The Web Fetch skill is a specialized tool designed for direct content retrieval when a target URL is already identified. It allows Openclaw Skills to bypass general search engines and interact directly with documentation, blog posts, or raw assets. By converting web pages into LLM-friendly formats like markdown or plain text, it enables precise summarization and data extraction while providing options for raw HTML inspection and direct image downloads.
This tool is particularly valuable for developers building automated workflows that require scraping specific data points or archiving web content. Unlike generic search tools, Web Fetch focuses on high-fidelity retrieval of known targets, ensuring that the AI agent receives the exact data needed without the noise of search engine results.
Web Fetch Use Cases
- Fetching and summarizing specific documentation or articles provided by a user.
- Extracting clean markdown from blog posts to facilitate long-form content analysis.
- Inspecting raw HTML source code to debug page structure, metadata, or embedded links.
- Downloading remote images directly to a local path for secondary processing or inspection.
How Web Fetch Works
- The AI agent identifies a specific URL provided by the user or identified in a previous workflow step.
- A Node.js command is executed using the fetch.js helper, specifying the desired output format and optional parameters like timeout.
- The skill attempts a secure HTTPS connection first, automatically falling back to HTTP if the initial request fails.
- Content is processed and converted based on the requested format, such as stripping boilerplate HTML for a clean markdown output.
- The skill returns a structured block containing metadata and the processed content for the agent to use in its response.
Web Fetch Setup
To integrate this skill into your environment, ensure you have the local fetch.js helper available in your project directory. Use the following commands to get started:
# Fetch a page as markdown (default)
node ./fetch.js --url "https://example.com"
# Fetch a page with a specific format and custom timeout
node ./fetch.js --url "https://example.com/docs" --format text --timeout 45
# Download an image to a local directory
node ./fetch.js --url "https://example.com/image.png" --output ./downloads/image.png
Web Fetch Data Schema & Taxonomy
The skill returns a structured response block that categorizes data based on the content type. This metadata is essential for Openclaw Skills to interpret the source correctly.
| Field | Description | Response Type |
|---|---|---|
| title | The page title extracted from HTML metadata | Text/HTML |
| url | The final URL after any redirects | All |
| mime | The MIME type of the retrieved asset | All |
| format | The output format requested (markdown, text, or html) | Text/HTML |
| content | The processed body of the web page | Text/HTML |
| image | The local file path where the image was saved | Image |
Web Fetch Advanced Features
- Automatic conversion of complex web layouts into clean, LLM-optimized markdown for better context window management.
- Built-in retry logic for Cloudflare 403 challenge responses using browser-like header emulation.
- Intelligent SVG handling that treats vector graphics as inspectable text rather than binary attachments.
- Large file protection that automatically rejects responses exceeding 5MB to preserve system performance.
- Configurable timeout settings ranging from 30 up to 120 seconds to handle slow or heavy web pages.
SKILL.md
Loading
Related Openclaw Skills
Sample Size (Basic)
A specialized statistical tool for estimating required sample sizes for clinical research and preliminary study planning.
gstack-openclaw-skills
A professional suite of 15 AI-driven development tools based on Y Combinator CEO Garry Tan's gstack, optimized for high-volume production code generation.
Web Search
A discovery-focused search tool that enables AI agents to find documentation, news, and external sources via Exa's hosted MCP endpoint.
Investment Five Questions Framework
A systematic value investing framework that uses five core questions to screen high-quality investment targets based on the Buffett-Munger philosophy.
ScRNA Cell Type Annotator
An automated bioinformatics tool for identifying and labeling cell clusters in single-cell RNA datasets based on marker gene expression.
A-Share Financial Analysis Assistant
A professional financial analysis tool that leverages remote agents to provide deep insights into A-share market stocks.
Featured*
