Web Distiller converts raw webpage URLs into clean, LLM-optimized Markdown or text to streamline AI agent data ingestion.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install web-distiller
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install web-distiller using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Web Distiller?
Web Distiller is a specialized utility designed to bypass the noise of raw HTML and deliver high-quality, cleaned content directly to your AI agents. By utilizing the Distiller API, this skill ensures that agents receive structured Markdown instead of cluttered webpage fetches, significantly reducing prompt token usage and improving context accuracy. As part of the Openclaw Skills library, it prioritizes reliability and developer experience by providing fallback mechanisms and clear setup paths.
The skill acts as a bridge between the live web and your AI workflows, offering flexible endpoints tailored to different needs. Whether you require a simple text summary or a complex distillation of a JavaScript-heavy site, Web Distiller handles the heavy lifting of content extraction and formatting.
Web Distiller Use Cases
- Extracting clean documentation from technical websites for RAG (Retrieval-Augmented Generation) systems.
- Automating bulk data collection from multiple URLs for competitive research or market analysis.
- Reducing token costs by stripping HTML boilerplate and ads before passing content to an LLM.
- Scouring JavaScript-heavy single-page applications using headless browser rendering options.
How Web Distiller Works
- The operator provides a target URL and specifies the desired output format (Markdown, Text, or JSON).
- The skill checks for a valid DISTILLER_API_KEY; if missing, it guides the user through the authentication process at webdistiller.dev.
- A request is sent to the preferred endpoint, defaulting to
/markdownfor broad compatibility or/distillfor advanced cleaning on paid plans. - The Distiller API processes the page, removing noise and formatting the core content.
- The cleaned data is returned to the agent workflow, ready for immediate processing or storage.
Web Distiller Setup
Install the package via pip:
pip install web-distiller
Configure your environment variables in your .env file or shell:
export DISTILLER_API_BASE=https://webdistiller.dev
export DISTILLER_API_KEY=your-api-key
If you do not have a key, sign in at [https://webdistiller.dev/signin](https://webdistiller.dev/signin) and retrieve it from your dashboard.
Web Distiller Data Schema & Taxonomy
Web Distiller supports multiple output schemas to fit different technical requirements:
| Format | Description | Best Use Case |
|---|---|---|
| Markdown | Structured content with headers and links. | LLM Prompts & Documentation |
| Text | Raw plain text with minimal formatting. | Smallest prompt payload |
| JSON | Full object including metadata and billing info. | Tool integration & Metadata tracking |
For batch operations, the skill returns a job_id which can be used to poll the status and retrieve a collection of processed results.
Web Distiller Advanced Features
- Multi-mode Batch Processing: Handle hundreds of URLs simultaneously using the
/batchendpoint. - Browser Rendering: Use the
--use-browserflag to execute JavaScript for dynamic web applications. - Dynamic Fallbacks: Automatically reverts from
/distillto/markdownif subscription limits are met to ensure workflow continuity. - Flexible Endpoints: Native support for both free-tier and premium extraction modes tailored for the Openclaw Skills framework.
SKILL.md
Loading
Related Openclaw Skills
PDF Processing Skill
A comprehensive toolkit for programmatic PDF manipulation, including text extraction, merging, splitting, and OCR capabilities.
Seedance AI Video Generator
A powerful AI video generation skill leveraging ByteDance Seedance models to transform text prompts and images into high-quality cinematic clips with audio.
WeChat Mini Program Automation Testing
A comprehensive automation toolkit for controlling WeChat DevTools to perform UI testing, navigation, and log analysis.
Multisig
A comprehensive CLI-based reference tool for managing and troubleshooting multi-signature wallet configurations in blockchain environments.
Teaching Materials Skill
An intelligent document generation tool for K-12 teachers to create high-quality courseware, lesson designs, and student task sheets automatically.
Google Research Pro
An advanced web-searching skill that uses Playwright to bypass bot detection and deliver technical research directly to Telegram.
Featured*
