A polite, remote MCP and REST service that converts public static HTML web pages into clean, LLM-friendly markdown.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install agent-web
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install agent-web using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is URL to Markdown?
The URL to Markdown skill (agent-web) is a specialized tool designed to fetch any public web page and convert it into clean, structured, and LLM-ready markdown format. This utility is perfect for developers building AI agents and workflows who need to ingest article text, extract documentation, or analyze online resources without dealing with noisy HTML parser pipelines. As an essential part of the Openclaw Skills repository, it provides seamless web extraction capabilities directly optimized for language model consumption.
This service operates under a 'charter-clean' polite philosophy: it strictly honors robots.txt constraints, identifies itself honestly without browser spoofing, and operates on a read-only GET basis. By integrating this service into your setup, you ensure safe, reliable, and compliant access to public static web content.
URL to Markdown Use Cases
- Summarizing articles or extracting core text from a specific URL.
- Feeding documentation pages into an AI agent without sending bloated, noisy HTML.
- Querying or performing analysis on the content of a remote web page.
- Generating low-token previews of websites to evaluate content before performing full fetches.
How URL to Markdown Works
- The user or AI agent invokes the skill via the
read_urlMCP tool or a direct GET REST endpoint with a target web URL. - The system checks the target host against safety rules, preventing SSRF attacks by blocking private, loopback, or internal IP addresses.
- The service parses and honors the site's robots.txt rules; if disallowed, the request is immediately refused.
- The service fetches the single URL with a polite, read-only GET request under a strict 2 MB body cap and 12-second timeout window.
- Unnecessary HTML, scripts, tags, and styling are stripped away to compile a clean, simplified markdown payload.
- The service returns a structured response containing the page title, word count, and optimized Markdown text.
URL to Markdown Setup
Integrating URL to Markdown with your agents and systems is straightforward. Since it runs as a remote MCP server, you can link your agents directly to the hosted endpoint or use standard tools like curl for REST interactions.
Remote MCP Integration
Add the following endpoint to your MCP-compatible client configuration:
- MCP Endpoint:
https://agent-web.foomworks.workers.dev/mcp
CLI Quick Start
You can use curl to interact with either the REST endpoints or the MCP JSON-RPC protocol. Here is how you can use Openclaw Skills resources via bash:
BASE=https://agent-web.foomworks.workers.dev
# Fetch clean markdown using the REST API
curl -s "$BASE/read?url=https://example.com/"
# Query the remote MCP server via a POST request
curl -s -X POST "$BASE/mcp" -H 'content-type: application/json' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"read_url","arguments":{"url":"https://example.com/"}}}'
URL to Markdown Data Schema & Taxonomy
REST Response Format
When querying the endpoint GET /read?url=<URL>, the service responds with a JSON object structured as follows:
| Property | Type | Description |
|---|---|---|
title |
String | The parsed title of the target web page |
words |
Number | Total word count of the extracted text |
markdown |
String | Clean, formatted LLM-ready markdown content |
Available MCP Tools
read_url: Fetches the full content of the requested URL. Accepts a single argument:url(string).read_url_preview: Returns only the first ~600 characters of the markdown content to preserve token counts.
Discovery Manifests
- MCP Descriptor:
GET /.well-known/mcp.json - Agent-to-Agent (A2A) Manifest:
GET /.well-known/agent-card.json - OpenAPI Specification:
GET /openapi.json
URL to Markdown Advanced Features
- Robots.txt Compliance: Automatically processes and respects the remote site's robots.txt configurations to prevent unauthorized scraping.
- Strict SSRF Protection: Every redirect hop is re-validated to prevent access to private, link-local, or loopback network targets.
- Token-Saving Previews: Includes the
read_url_previewendpoint and tool, retrieving just the first 600 characters to evaluate context costs before pulling the full payload. - Open Standards Support: Includes native integration hooks for OpenAPI 3.1, Agent-to-Agent (A2A), and JSON-RPC 2.0 (MCP).
- Zero-Config Deployment: Run instantly with no API keys, accounts, or payment gateways needed.
SKILL.md
Loading
Related Openclaw Skills
Prompting Modes
An advanced reasoning toolbox featuring five proven prompting paradigms—Chain-of-Thought, Self-Consistency, Tree-of-Thought, ReAct, and Plan-and-Execute—optimized as highly adaptable Openclaw Skills for autonomous AI agents.
Budget Performance Management Workflow Skill
An automated 7-step pipeline designed for end-to-end financial budget performance auditing, efficiency tracking, and comprehensive reporting.
Bidding Audit Workflow
A comprehensive 9-step audit workflow designed to detect bid-rigging and collusion risks through graph association, price pattern analysis, and similarity checks.
Economic Responsibility Audit Workflow
An automated 10-step auditing pipeline for leadership economic responsibility audits, covering everything from financial analysis to final report compilation.
Work Productivity Obsidian Vaults Workflow Helper
A local-first productivity tool designed to automate, secure, and fix bugs in Obsidian-style markdown vaults.
Software Data GitHub Interact Developer Helper
A developer-focused AI assistant skill designed to streamline GitHub-style workflows, resolve bugs, secure repository setup, and build reliable local automation tools.
Featured*
