A specialized tool for extracting clean text or raw HTML from any webpage to power AI-driven research and automation.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install desearch-crawl
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install desearch-crawl using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Desearch Crawl?
Desearch Crawl is an essential utility within the ecosystem of Openclaw Skills that allows developers and AI agents to programmatically retrieve webpage content. By simplifying complex web pages into readable text, it ensures that LLMs receive high-quality data without the noise of scripts and styling. This skill is optimized for speed and reliability, making it a cornerstone for any agentic workflow that requires real-time information retrieval from the web.
Built to work seamlessly with modern AI development environments, this tool leverages the Desearch API to bypass common scraping hurdles. Whether you are building a research assistant or a technical documentation reader, integrating this into your Openclaw Skills collection provides a direct bridge to live web data, formatted specifically for machine consumption.
Desearch Crawl Use Cases
- Reading technical documentation pages to provide context for coding tasks.
- Scraping news articles or blog posts for automated summary generation.
- Retrieving raw HTML for detailed page structure analysis and debugging.
- Feeding real-time web content into Openclaw Skills for enhanced research capabilities.
How Desearch Crawl Works
- The agent or user specifies a target URL and chooses between text or HTML output formats.
- The script authenticates with the Desearch platform using a secure API key.
- Desearch processes the URL, rendering the page and extracting the requested content type.
- The output is returned as a plain string (text) or markup (HTML) to be used by the calling Openclaw Skills agent.
Desearch Crawl Setup
To get started with this skill, follow these installation steps:
- Obtain an API key from the Desearch Console at https://console.desearch.ai.
- Configure your environment variable:
export DESEARCH_API_KEY='your-key-here'
- Execute the crawl script using the following command structure:
# Default text crawl
scripts/desearch.py crawl "https://example.com"
# HTML format crawl
scripts/desearch.py crawl "https://example.com" --crawl-format html
Desearch Crawl Data Schema & Taxonomy
The skill prioritizes minimal token usage by returning direct data rather than nested objects. This makes it highly efficient for Openclaw Skills implementations.
| Format | Output Description |
|---|---|
| text | Default. Returns the core readable text of the webpage, stripped of HTML tags. |
| html | Returns the full, raw HTML source of the target page for structural analysis. |
| Error (401) | Returns a JSON object indicating an invalid or missing API key. |
| Error (402) | Returns a JSON object indicating insufficient account balance. |
Desearch Crawl Advanced Features
- Configurable output formats allowing for either text-only or full-source HTML extraction.
- Lightweight footprint designed to prevent context window bloat in Openclaw Skills agents.
- Robust error handling for authentication and billing status through standard JSON responses.
- Direct integration with the Desearch AI infrastructure for high-fidelity rendering of dynamic pages.
SKILL.md
Loading
Related Openclaw Skills
AI Search By Desearch
An AI-powered multi-source search engine that aggregates, filters, and summarizes content from across the web and social platforms.
ClawOps
A central orchestration layer that manages and coordinates multiple Openclaw Skills to build unified, intelligent automation workflows.
Plan2Meal Skill
An AI-powered skill for managing Plan2Meal recipes and grocery lists through natural language commands.
YouTube Music Control
A natural language interface for YouTube Music that leverages browser automation for hands-free playback and library management.
Desearch Web Search
A high-performance web search skill that delivers real-time search engine results and structured snippets to AI agents.
Desearch X (Twitter) Search
A real-time X (Twitter) search and monitoring tool for AI agents to find posts, track users, and analyze social engagement.
METADATA
Featured*
