A high-performance web and social media reader that converts URLs into AI-ready Markdown without requiring external API keys.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install deepreader-skill
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install deepreader-skill using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is DeepReader?
DeepReader serves as the essential content ingestion layer for agents using Openclaw Skills. It specializes in transforming unstructured data from diverse sources like social media and video platforms into structured Markdown files. By utilizing specialized parsers, it bypasses the need for complex API integrations for platforms like X or Reddit, ensuring your agents have immediate access to the latest web data.
This skill is designed to bridge the gap between the live web and an agent's long-term memory, facilitating better context and informed decision-making. As part of the broader ecosystem of Openclaw Skills, it provides a seamless way to feed real-time information into any AI workflow.
DeepReader Use Cases
- Ingesting full Twitter threads or X articles into agent context for social listening.
- Extracting Reddit discussions and top-level comments for community sentiment analysis.
- Fetching automated YouTube transcripts for video summarization and knowledge extraction.
- Scraping blog posts, technical documentation, or news articles to build local knowledge bases.
- Batch processing multiple URLs from a single user prompt for rapid multi-source research.
How DeepReader Works
- The system automatically scans user messages to detect one or more URLs using regex patterns.
- A routing logic identifies the source platform, such as Twitter, Reddit, YouTube, or generic web sources.
- Specialized parsers fetch the content using optimized methods, such as the FxTwitter API or the Reddit .json suffix.
- The raw content is processed and cleaned to remove noise like advertisements, navigation menus, and headers.
- The resulting clean text is formatted into a standardized Markdown file containing detailed YAML metadata.
- The processed file is automatically saved to the designated memory path for the agent to access.
DeepReader Setup
To get started with this component of your Openclaw Skills setup, ensure your environment has the necessary parsing libraries installed.
# Install the core parsing dependencies
pip install trafilatura youtube-transcript-api beautifulsoup4
Next, configure your storage directory to define where ingested content should be stored:
# Set the environment variable for memory storage
export DEEPREEDER_MEMORY_PATH="../../memory/inbox/"
DeepReader Data Schema & Taxonomy
DeepReader generates structured Markdown files. Each file includes a YAML frontmatter block designed to work efficiently with other Openclaw Skills for indexing and retrieval.
| Metadata Field | Description |
|---|---|
| title | The title of the post, video, or webpage |
| source_url | The original URL that was processed |
| domain | The source domain (e.g., x.com, youtube.com) |
| parser | The specific extraction engine used |
| ingested_at | ISO 8601 timestamp of when the content was read |
| content_hash | SHA256 hash used for content deduplication |
| word_count | The total word count of the extracted content |
DeepReader Advanced Features
- Automatic fallback to Nitter for Twitter content to ensure high availability without official API keys.
- Native support for batch URL processing, allowing an agent to ingest an entire bibliography in one pass.
- Intelligent transcript extraction for YouTube videos across multiple languages.
- Zero-config integration with the default Openclaw Skills memory architecture.
- Customizable logging levels to monitor extraction success and troubleshoot parser selection.
SKILL.md
Loading
Related Openclaw Skills
DeepBook CLI Skill
A powerful command-line interface for end-to-end DeepBook operations including market data, account management, and on-chain execution.
Quality Convergence Engine
A multi-dimensional engine that deconstructs requirements into objective acceptance and failure criteria to eliminate implementation defects.
Portainer
Manage Docker infrastructure and git-based stacks through the Portainer REST API using simple CLI commands or AI agent interactions.
PM2 Process Manager
A production-grade process manager for Node.js applications with built-in load balancing and auto-restart capabilities.
DocuClaw
A sovereign document intelligence and archival system that extracts structured data from files 100% locally using AI.
OpenClaw Reference Manual
A complete technical reference for managing OpenClaw configurations, CLI operations, and multi-channel AI agent deployments.
Featured*
