Smart Fetch is a high-performance web ingestion tool designed to deliver token-efficient, markdown-optimized content directly to AI agents while enforcing strict safety and size constraints.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install smart-fetch
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install smart-fetch using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Smart Fetch?
Smart Fetch is a specialized utility within the Openclaw Skills ecosystem designed to solve the noise and token-bloat issues associated with traditional web scraping. By prioritizing markdown-first content negotiation, it ensures that AI agents receive clean, structured data without the overhead of unnecessary HTML boilerplate. When markdown is unavailable, the tool leverages advanced Readability and Turndown fallbacks to extract only the most relevant content.
Beyond simple extraction, Smart Fetch acts as a security and quality firewall. It analyzes fetched content for potential threats, such as shell execution lures or API key requests, and provides actionable metadata. This allows developers to build more resilient AI workflows that can automatically decide whether to retry a request, skip summarization, or flag content for manual review based on the provided quality and safety signals.
Smart Fetch Use Cases
- Extracting clean article, documentation, or blog content for LLM context windows.
- Implementing token-efficient web research within Openclaw Skills automated workflows.
- Protecting AI agents from malicious prompts or 'jailbreak' lures hidden in external web pages.
- Standardizing web data ingestion across different domains using a unified, cache-aware CLI tool.
- Handling high-volume scraping tasks where strict memory and character limits are required to prevent system crashes.
How Smart Fetch Works
- The tool initiates a request with a preferred header of
text/markdown, attempting to fetch pre-formatted content. - If the server provides markdown, it is returned directly; if HTML is provided, the tool processes the page using Readability to strip noise and Turndown to convert it to markdown.
- Post-extraction filters are applied to enforce user-defined character and byte limits on the final body content.
- A comprehensive safety scan is performed to identify malicious command lures or suspicious data requests within the content.
- The system generates a structured response including the sanitized body, metadata, and a recommended next action for the AI agent.
Smart Fetch Setup
To start using this utility in your Openclaw Skills environment, ensure you have Node.js installed and run the following command:
# Basic installation and usage
node index.js <url>
# Recommended setup with caching and size limits
node index.js --max-chars 12000 --cache-ttl 3600 <url>
You can also configure persistent environment variables for your Openclaw Skills deployment:
export SMART_FETCH_MAX_CHARS=15000
export SMART_FETCH_CACHE_DIR="./.cache/smart-fetch"
Smart Fetch Data Schema & Taxonomy
Smart Fetch organizes output into a structured format that prioritizes metadata for automated routing. The schema includes:
| Field | Description |
|---|---|
body |
The extracted markdown content |
metadata |
Includes path, title, and domain info |
severity |
Status of the fetch (info, warn, error) |
warnings |
Quality flags like body_too_short or missing_title |
safetyFlags |
Detection of contains_shell_exec_lure or contains_api_key_request |
recommendedNextAction |
Guidance for the agent (e.g., manual_review_needed) |
Smart Fetch Advanced Features
- Advanced Content Negotiation: Automatically handles the transition between markdown and HTML sources.
- Exponential Backoff: Built-in retry logic with configurable intervals for unstable network conditions.
- Domain Governance: Strict allowlist and blocklist controls to manage where your Openclaw Skills are permitted to crawl.
- Unicode Accuracy: Character limits are calculated using Unicode codepoints rather than simple bytes to ensure precise context window management.
- Safety Contract: Hardened security layer that treats all fetched data as untrusted, preventing accidental script execution during the ingestion phase.
SKILL.md
Loading
Related Openclaw Skills
Central Model Switch & Rescue
A mission-critical utility for managing model failover and node recovery within the OpenClaw ecosystem.
Ontology + Causal Enhanced
A sophisticated dual-engine skill that combines structured knowledge graphs with causal reasoning for advanced AI decision-making.
AI Persona Engine
A framework for building emotionally intelligent AI personas using actor-direction and the Doctrine of Embodied Intelligence rather than rigid technical checklists.
Website Flow Monitor
An automated system for discovering business-critical user flows and scheduling recurring synthetic health checks to ensure website uptime and functional integrity.
Wealth Builder
Wealth Builder is a comprehensive personal finance and investment engine designed to help users build, protect, and grow wealth through structured planning and disciplined execution.
Go Mode — Autonomous Goal Execution
Go Mode is an autonomous execution framework that transforms high-level goals into actionable, multi-step plans with built-in human oversight.
Featured*
