A highly structured URL extraction skill converting web pages into clean markdown through a deterministic fallback pipeline of markdown.new and Jina Reader.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install web-to-md
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install web-to-md using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Web to Markdown?
The Web to Markdown skill is a robust, console-first content extraction workflow optimized for AI agents. Rather than relying on unpredictable, open-ended web browsing that consumes excessive tokens and returns noisy HTML structures, this tool enforces a strict, step-by-step fallback chain to convert specific user-provided URLs into readable Markdown. This ensures that downstream language models process only clean, highly structured textual data.
As a foundational utility within the suite of Openclaw Skills, this skill maximizes content quality without incurring the heavy performance costs of browser spin-ups. By leveraging specialized API endpoints like markdown.new and r.jina.ai before falling back to local tools, it guarantees consistent web parsing across diverse web targets.
Web to Markdown Use Cases
- Parsing and summarizing specific blog posts, documentation pages, or articles provided directly by users.
- Generating high-quality text corpora from reference URLs for RAG (Retrieval-Augmented Generation) ingestion pipelines.
- Reducing token consumption and noisy web junk (ads, navigations, footers) before transmitting content to LLMs.
- Executing predictable, CLI-native web extraction workflows without managing complex headless browser infrastructure.
How Web to Markdown Works
- Scope Gating: The workflow evaluates if the target URL has been explicitly provided, bypassing broad-scoped web searches.
- URL Normalization: Normalizes input URLs to ensure safe, quoted execution in standard shells.
- AI-Mode Extraction: Queries markdown.new using the AI method as the primary extraction mechanism.
- Auto-Mode Fallback: Automatically falls back to markdown.new Auto method if the AI parser returns insufficient text or times out.
- Engine-Mode Fallback: Escalates to Jina Reader with a browser header configuration if both previous methods fail.
- Quality Gate Check: Compares output against strict heuristic criteria (e.g., token length, completeness, missing sections) to detect truncation.
- Native Tooling Escapes: Defaults to local agent browser engines only when all cloud-based converters fail.
- Lineage Attribution: Affixes a standard provenance label to the final response to record which pipeline tier succeeded.
Web to Markdown Setup
To leverage this skill within your Openclaw Skills configuration, ensure the system environment contains curl. No specialized API keys are strictly required to begin routing requests.
Verify that curl is available in your shell environment:
which curl
Configure your AI agent execution blocks to orchestrate the fallback loop programmatically, starting with the primary endpoint:
curl -s "https://markdown.new/{URL}?method=ai"
Web to Markdown Data Schema & Taxonomy
Extraction Provenance Taxonomy
Each successful run produces a structured provenance label, making it easy to track source extraction origin:
| Label | Description / Condition |
|---|---|
markdown.new:ai |
Content extracted successfully via the markdown.new AI service. |
markdown.new:auto |
Content extracted via markdown.new Auto-mode after the AI parser failed. |
r.jina.ai |
Content extracted using the Jina Reader engine after both primary targets timed out or failed. |
agent-tools |
Content extracted utilizing the agent's native fallbacks due to complete API failure. |
Content Quality Evaluation Criteria
Extracted content is programmatically evaluated based on the following metrics before acceptance:
- Character Count Threshold: A 1,200-character soft heuristic is used to catch truncated articles.
- Structural Validity: Rejects pages missing critical elements (main text blocks, requested documentation sections).
- Boilerplate Filtering: Identifies and blocks pages dominated by login walls, cookie notices, or navigation templates.
Web to Markdown Advanced Features
- Strict Determinism: Replaces unpredictable browser-led scraping with structured, cost-efficient fallback loops.
- Built-in Quality Control: Employs heuristic analysis to block corrupted, truncated, or heavily gated content automatically.
- Multi-URL Handling: Orchestrates parallel, isolated fallback queries when processing multiple links in a single request.
- Shell Injection Protection: Ensures safe token handling and URL quoting to protect local execution runtimes.
- Source Traceability: Seamlessly interfaces with Openclaw Skills workflows to provide verifiable data lineage for downstream LLM agents.
SKILL.md
Loading
Related Openclaw Skills
GIF Multi
A cross-platform GIF reaction skill that enables AI agents to search and send optimized animated GIFs on Telegram, WhatsApp, Discord, Signal, and other messaging channels.
Text to SQL
An AI-powered skill that bridges the gap between natural language and structured database queries by translating plain English into precise, schema-validated SQL.
AI Image To Code
Converts user interface screenshots into accurate, functional HTML/CSS or React and Tailwind CSS components using advanced vision models.
Quick Note
A lightweight, local-first note-taking tool that instantly captures ideas, to-dos, and reading lists with automatic categorization and zero external dependencies.
Text to SQL
A developer-focused agent skill that converts natural language requests into schema-accurate, dialect-specific SQL queries.
AI Image To Code
An advanced AI-powered utility that converts UI screenshots and layout designs into fully functional, responsive HTML/CSS or React + Tailwind CSS code.
Featured*
