A comprehensive web scraping and search utility that provides AI agents with structured data from Google, Amazon, YouTube, and Reddit.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install decodo-scraper-skill
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install decodo-scraper-skill using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Decodo Scraper?
The Decodo Scraper is a versatile tool designed to empower AI agents with the ability to retrieve and process real-time web data. As part of the Openclaw Skills ecosystem, it leverages the Decodo Web Scraping API to transform unstructured web content into LLM-friendly formats like Markdown and structured JSON. This skill is essential for developers building agents that require deep market research, product monitoring, or sentiment analysis across major platforms.
By integrating this skill, users can bypass the complexities of manual scraping and focus on data analysis. Whether you are fetching AI-generated overviews from Google or extracting detailed product specifications from Amazon, this skill ensures that your Openclaw Skills setup remains data-rich and context-aware.
Decodo Scraper Use Cases
- Conducting automated market research by extracting Google organic search results and AI overviews.
- Monitoring e-commerce trends and competitor pricing by scraping Amazon product pages and search listings.
- Summarizing video content by fetching transcripts through the YouTube subtitles tool.
- Analyzing community trends and public sentiment by extracting data from Reddit posts and subreddits.
- Gathering clean, Markdown-formatted web content for RAG (Retrieval-Augmented Generation) pipelines.
How Decodo Scraper Works
- The user initiates a command specifying a target platform (e.g., google_search, amazon, or reddit) and a query or URL.
- The script authenticates with the Decodo API using the DECODO_AUTH_TOKEN provided in the environment.
- The scraper sends the request to the Decodo backend, which handles proxy rotation, headless browsing, and CAPTCHA solving.
- The API returns either parsed JSON (for structured sites like Amazon and Reddit) or clean Markdown (for universal web scraping).
- The Openclaw Skills agent receives the processed data for further analysis or response generation.
Decodo Scraper Setup
To use this skill, you must first obtain a Decodo Auth Token from your Decodo Dashboard.
Set your credentials in your environment:
export DECODO_AUTH_TOKEN='your_basic_auth_token'
Basic command structure:
python3 tools/scrape.py --target [TARGET_NAME] --query "[SEARCH_QUERY]" --url "[URL]"
Decodo Scraper Data Schema & Taxonomy
The skill returns data tailored to the specific target platform, optimizing for clarity and token efficiency.
| Target Platform | Output Format | Key Data Points |
|---|---|---|
| Google Search | JSON | organic results, ai_overviews, related_questions, forum discussions |
| Universal URL | Markdown | Cleaned page text, headings, and links |
| Amazon | JSON | Product titles, ads, pricing, and delivery info |
| YouTube | Text | Full video transcripts and subtitles |
| JSON | Post content, subreddit listings, and thread metadata |
Decodo Scraper Advanced Features
- Geographic targeting for Google searches using the
--geoflag to see results from specific regions. - Locale-specific results using the
--localeparameter for language-sensitive research. - Native support for Google AI Overviews, allowing agents to see AI-synthesized answers directly.
- Flexible output modes, including Markdown, to ensure compatibility with various Openclaw Skills processing modules.
- Robust error handling that outputs JSON-formatted error messages to stderr for automated debugging.
SKILL.md
Loading
Related Openclaw Skills
Repo Analyzer
A comprehensive GitHub repository auditing tool that provides automated trust scores and security due diligence.
Cross-Model Review
An adversarial review system that uses two different AI models to stress-test and refine complex implementation plans autonomously.
Calendly Quick Book
A natural language booking tool that schedules Calendly meetings instantly without manual link sharing or UI navigation.
dm-bot Agent Messaging
A secure, encrypted messaging platform specifically designed for AI agents to interact via DMs, group chats, and public posts.
Decodo Scraper
A high-performance web scraping and data extraction tool for Google, Amazon, YouTube, and Reddit.
Eastmoney Stock Query
A specialized financial data skill for retrieving real-time stock market information, price fluctuations, and trading volumes from Eastmoney.
Featured*
