A lightweight, zero-dependency Python tool for fetching web content and converting HTML to markdown securely.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install url-fetcher
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install url-fetcher using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is URL Fetcher?
URL Fetcher is a streamlined utility designed for Openclaw Skills that enables autonomous agents to retrieve web content using only the Python standard library. By bypassing the need for external dependencies or expensive third-party APIs, it provides a cost-effective solution for automated data collection.
This skill prioritizes security through strict URL and file path validation, ensuring that agents operate within safe boundaries while gathering information from the web. It is particularly effective for developers who need a reliable way to ingest web data without the overhead of heavy scraping frameworks or the recurring costs of paid extraction services.
URL Fetcher Use Cases
- Research collection for saving articles and blog posts locally for agent processing.
- Content aggregation from multiple sources to feed into data analytics or LLM workflows.
- Simple web scraping when a full browser environment or paid API is unnecessary.
- Converting static HTML pages into basic markdown format for better readability by AI agents.
How URL Fetcher Works
- The user provides a target URL and an optional output file path via the CLI.
- The skill validates the URL to ensure it is not targeting internal networks (like localhost) or restricted protocols.
- Content is retrieved using the Python urllib library with a default 10-second timeout.
- If requested, the HTML is processed via regex-based logic to produce a simplified markdown version.
- The skill checks the destination path against safety rules to prevent unauthorized system writes.
- The final content is either displayed in the terminal or saved to the validated file location.
URL Fetcher Setup
This utility is ready to use within your Openclaw Skills directory. No pip installation is required. Use the following commands to get started:
# Fetch and preview content in the terminal
url_fetcher.py fetch https://example.com
# Fetch and save HTML to a local file
url_fetcher.py fetch https://example.com ~/workspace/page.html
# Fetch and convert to basic markdown
url_fetcher.py fetch --markdown https://example.com ~/workspace/page.md
URL Fetcher Data Schema & Taxonomy
The skill manages data primarily through local file outputs and terminal streams. It follows a strict path validation schema to ensure security.
| Item | Description | Format |
|---|---|---|
| Fetched Content | Raw HTML or text from the source URL | .html / .txt |
| Converted Output | Basic markdown version of the HTML | .md |
| Allowed Paths | Restricted to ~/workspace, home, or /tmp | String |
| Blocked Paths | System paths (e.g., /etc) and sensitive dotfiles | N/A |
URL Fetcher Advanced Features
- Integration with other Openclaw Skills like research-assistant for organized knowledge management.
- Support for batch fetching scripts to handle multiple URLs sequentially using simple bash loops.
- Built-in security hooks that prevent Server-Side Request Forgery (SSRF) by blocking internal IPs and sensitive protocols.
- Zero-cost operation with no external package requirements, making it ideal for budget-constrained autonomous agents.
SKILL.md
Loading
Related Openclaw Skills
Task Runner
A persistent task management system designed to track projects and todo lists across multiple AI agent sessions.
Security Checker
A specialized static analysis tool designed to scan Python-based skills for security vulnerabilities and hardcoded credentials before deployment.
Seasonal Planting Guide
A technical command-line tool for generating region-specific planting calendars and managing botanical data based on USDA Hardiness Zones.
Research Assistant
A sophisticated knowledge management tool for agents to structure, store, and retrieve research notes across multiple sessions.
A-Share Glossary Tutor
A structured tutor and note-taking assistant for mastering Chinese A-share financial indicators and market terminology.
A-Share Metrics Card
An AI-driven tool for generating standardized financial health reports and key metrics for Chinese A-share stocks.
Featured*
