A secure web fetching utility that intelligently routes public URLs through markdown cleaners while keeping sensitive or internal links local.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install safe-smart-web-fetch
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install safe-smart-web-fetch using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Safe Smart Web Fetch?
Safe Smart Web Fetch is a specialized tool designed to enhance data privacy and extraction efficiency for AI agents. It intelligently analyzes target URLs to determine the safest fetching method, ensuring that internal network addresses or links containing authentication tokens are never exposed to third-party services. By integrating this into your Openclaw Skills workflow, you can automate the conversion of public web pages into clean, readable Markdown using industry-standard tools like Jina Reader, while maintaining strict local-only fetching for sensitive resources.
This skill provides a robust layer of security by filtering URLs based on query parameters, hostname, and page type. It acts as a smart proxy that balances the need for high-quality content extraction with the necessity of data sovereignty, making it an essential component for developers building privacy-conscious AI applications.
Safe Smart Web Fetch Use Cases
- Extracting documentation or articles from public websites into clean Markdown format.
- Securely fetching data from internal company dashboards, localhost services, or private network nodes.
- Preventing accidental leakage of API keys, session tokens, or reset links to third-party content cleaners.
- Building robust research agents within the Openclaw Skills ecosystem that require high-quality, noise-free web content.
How Safe Smart Web Fetch Works
- Analyze the target URL for sensitive patterns, including auth tokens, keys, signatures, and private keywords in the query or fragment.
- Verify the host to identify localhost addresses, internal IP ranges (10.x, 172.16-31.x, 192.168.x), or .local domains.
- Determine the routing logic: if the URL is identified as sensitive or internal, execute a direct local fetch to bypass third-party exposure.
- For public URLs, attempt a prioritized sequence of high-quality Markdown extraction services: Jina Reader, markdown.new, and defuddle.md.
- Fall back to a raw local fetch of the HTML content if third-party services fail or are intentionally bypassed for safety.
- Return a structured JSON response containing the processed content, the source used, and detailed safety metadata.
Safe Smart Web Fetch Setup
To use this skill within your environment, execute the fetch script via the command line. Ensure you have the necessary dependencies installed in your Python environment.
# Basic usage to fetch and clean a public article
python3 scripts/fetch.py "https://example.com/article"
# Fetch with JSON output for structured data processing
python3 scripts/fetch.py "https://example.com/article" --json
Safe Smart Web Fetch Data Schema & Taxonomy
When executed with the --json flag, the skill returns a structured object with the following fields:
| Field | Type | Description |
|---|---|---|
success |
Boolean | Indicates if the content was successfully retrieved. |
url |
String | The original URL that was processed. |
content |
String | The extracted Markdown or raw HTML content. |
source |
String | The specific service or method used (e.g., Jina, markdown.new, local). |
used_third_party |
Boolean | True if an external cleaning service was utilized. |
blocked_reason |
String | Explains why a third-party service was bypassed (if applicable). |
error |
String | Contains error details if the operation failed. |
Safe Smart Web Fetch Advanced Features
- Automatic sensitive parameter detection: Detects tokens, session IDs, and auth codes in URLs to prevent data leakage.
- SSRF Protection: Automatically identifies and restricts external routing for internal network addresses and private hosts.
- Multi-Stage Fallback: Implements a resilient chain of extraction services to ensure the highest quality Markdown output.
- Non-Invasive Integration: Designed to work alongside existing Openclaw Skills without modifying global configurations or built-in tools.
SKILL.md
Loading
Related Openclaw Skills
Amazon Listing Competitor Analysis
A dual-phase intelligence tool that extracts live Amazon product data and synthesizes strategic opportunity maps to outperform competitors.
MVP Strategy Agent
A strategic guide for building high-impact products with minimum effort by focusing on manual validation before automation.
Validate Idea
A strategic business advisor skill that helps you validate market demand and profitability before writing a single line of code.
Processize
Processize transforms your software concepts into repeatable, manual workflows that validate value before you write a single line of code.
JSON Deep Compare
A robust Python-based utility designed for deep recursive comparison of JSON structures with automated Excel reporting and intelligent array matching.
Marketing Plan AI Skill
An AI agent skill that generates minimalist, content-first marketing strategies focused on building an organic audience rather than buying ads.
Featured*
