A comprehensive web intelligence toolkit that gives AI agents one-click access to data from social media, video platforms, and search engines.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install web-scout
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install web-scout using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Web Scout?
Web Scout is a powerful data acquisition layer for AI agents, built upon the Agent Reach framework. It eliminates the friction of manual scraping or expensive API subscriptions by providing a unified interface to pull content from diverse sources like Twitter, Reddit, YouTube, and specialized platforms like Xiaohongshu and Boss Zhipin. By integrating these capabilities into Openclaw Skills, developers can build agents that are truly connected to the live web.
This skill focuses on accessibility and efficiency, utilizing open-source tools like yt-dlp and Jina Reader to process content into agent-friendly formats. Whether you need to monitor RSS feeds, extract video subtitles, or search GitHub repositories, Web Scout provides the necessary CLI tools and MCP servers to get the job done without complex overhead.
Web Scout Use Cases
- Automated social media monitoring and sentiment analysis across X and Reddit.
- Content summarization from video platforms like YouTube and Bilibili using subtitle extraction.
- Technical research and dependency tracking via GitHub CLI integration.
- Market research and job trend analysis using LinkedIn and Boss Zhipin data.
- Building real-time news aggregators using RSS and general web search via Exa.
How Web Scout Works
- The user installs the Agent Reach CLI and initializes the environment to handle dependencies automatically.
- Platform-specific configurations, such as browser cookies or proxies, are stored securely in a local configuration file.
- The AI agent invokes specific CLI commands or MCP tools (e.g., xreach or mcporter) based on the target platform.
- Web Scout fetches the raw data from the target site, bypassing standard API restrictions where possible using specialized scrapers.
- The collected data is cleaned and returned to the agent in structured JSON or Markdown format for immediate processing.
Web Scout Setup
First, install the core CLI directly from the source:
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
Next, automatically install all required dependencies to support your Openclaw Skills:
agent-reach install --env=auto
Verify the installation health and check for missing requirements:
agent-reach doctor
For platforms requiring authentication, such as Twitter, configure your session cookies:
agent-reach configure twitter-cookies "YOUR_COOKIE_STRING"
Web Scout Data Schema & Taxonomy
Web Scout handles data through various specialized tools, primarily outputting structured text for agent consumption.
| Source | Format | Typical Metadata |
|---|---|---|
| Social Media | JSON | Author, timestamp, text content, engagement stats |
| Video Sites | JSON/SRT | Title, description, subtitles, video duration |
| Web/Search | Markdown | Page title, URL, cleaned body text via Jina Reader |
| GitHub | JSON | Repo stats, issue descriptions, file trees |
All configurations and session data are stored locally at ~/.agent-reach/config.yaml with restricted file permissions.
Web Scout Advanced Features
- Safe Installation Mode: Use the
--safeflag to audit dependencies without making changes to the system. - Docker Containerization: Seamlessly run complex scrapers like Xiaohongshu-MCP via Docker to isolate browser environments.
- Global Proxy Configuration: Support for HTTP/HTTPS proxies to navigate regional blocks on platforms like Reddit or Bilibili.
- Automated Health Monitoring: Use the
watchcommand to ensure data collection tasks are functioning correctly for scheduled jobs. - Multi-Agent Compatibility: Easily integrates with MCP-compatible agents for distributed data gathering tasks.
SKILL.md
Loading
Related Openclaw Skills
AI News ZH
An automated pipeline that harvests global AI news, translates it into Chinese, and delivers curated daily reports to your preferred communication channels.
CoPAW Yindeng Notice and Result Crawler
An automated crawler and LLM-powered analysis tool for extracting structured data from Yindeng non-performing loan transfer announcements.
Axelrod
Axelrod is an AI-driven trading agent designed for the Base chain that executes complex DeFi operations and on-chain queries via natural language commands.
Wise
A comprehensive automation tool for managing international transfers, multi-currency accounts, and real-time exchange rates via AI agents.
ClawTunes
A social music platform where AI agents compose, share, and remix tunes using ABC notation.
Yahoo Finance CLI
A high-performance Python CLI for fetching real-time financial data, stock quotes, and company fundamentals directly from Yahoo Finance.
Featured*
