A high-performance web crawling and scraping skill designed for AI agents to retrieve structured Amazon product, search, and general public web page data instantly.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install open-crawl
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install open-crawl using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Open Crawl?
open-crawl is a powerful utility designed for AI agents to gather real-time data from public web pages, with deep native support for Amazon. By leveraging Claw School's hosted data API, this skill allows agents to fetch product details (prices, ratings, BSR, and reviews) or search engine result pages (SERPs) and receive them in clean, structured JSON format. This eliminates the need for agents to write custom scraping logic or parse complex HTML, boosting reliability and speed.
As part of the ecosystem of Openclaw Skills, this tool handles proxy rotation, JavaScript rendering, and anti-bot measures behind the scenes. This ensures that your agent can seamlessly query e-commerce data or general web pages without setting up expensive third-party scraping infrastructure.
Open Crawl Use Cases
- Monitoring live Amazon product prices, ratings, and Best Seller Ranks (BSR).
- Performing automated e-commerce market research and analysis by pulling search engine results pages (SERPs).
- Retrieving raw HTML content from public web pages for content extraction or summarization.
- Enhancing AI agent intelligence by providing up-to-date, live web data instead of outdated training data.
How Open Crawl Works
- The AI agent receives an explicit request to lookup public web or Amazon product data.
- The agent issues an HTTPS POST request to the hosted Claw School API endpoint ({CLAW_API_BASE}/api/scrape) authenticating via a standard Bearer token.
- The hosted API handles JavaScript rendering (using scraper mode) or direct proxy retrieval depending on the configured mode.
- Amazon pages are automatically parsed into structured JSON fields (such as ASIN, rating, bullet points, reviews).
- The agent receives the response, falling back to raw HTML if parsing quality is low, and presents the clean structured data directly to the user.
Open Crawl Setup
To deploy this skill with your AI agent, follow these steps to configure your credentials:
- Obtain a CLAW_KEY by registering at claw-school.com.
- Configure your agent's environment with the following environment variables (automatically handled during the package installation and redeem step) to integrate this capability into your suite of Openclaw Skills:
export CLAW_KEY="your_claw_key_here"
export CLAW_API_BASE="https://api.claw-school.com"
- Test the setup by sending a sample request via your agent's web-request capability.
Open Crawl Data Schema & Taxonomy
The API returns structured JSON data optimized for direct consumption by your AI agent. Depending on the target URL, the response schema adapts to provide the most relevant information:
Amazon Product Page (PDP) Schema
| Field | Type | Description |
|---|---|---|
| url | String | Target product page URL |
| status | Integer | HTTP response status code |
| parsed.asin | String | Amazon Standard Identification Number |
| parsed.title | String | Title of the product listing |
| parsed.price | String | Current display price |
| parsed.rating | String | Average star rating |
| parsed.review_count | String | Total number of customer reviews |
| parsed.bsr | Array | Best Seller Rank categories |
| parsed.reviews | Array | Sample of customer reviews with author, rating, title, and body |
General Web Page Fallback
For non-Amazon public web pages, the response contains raw HTML in the body field with the parsed object set to null, allowing custom agent-side parsing.
Open Crawl Advanced Features
- Dual Scrape Modes: Seamlessly switch between "scraper" mode (with full JS-rendering for interactive pages like Amazon) and "proxy" mode (fast raw HTTP for static sites).
- Geo-Targeted Crawling: Specify a two-letter country code (e.g., "US", "JP") to scrape location-specific Amazon localized pricing and regional listings.
- Custom JS Snippets: Execute custom JavaScript scripts after page load (in scraper mode) to interact with the DOM or extract custom elements.
- Built-in Error Resolution: Returns structured error codes (e.g., QUOTA_EXCEEDED, URL_NOT_ALLOWED) alongside action directives that agents can relay directly to users.
- Automatic Fallback Mechanism: Includes raw HTML (body) inside the response if structured extraction quality drops below thresholds.
SKILL.md
Loading
Related Openclaw Skills
Google Routes
An AI-ready wrapper for Google Routes that enables autonomous routing and distance matrix computation via a secure CLI connector.
Cruise Package Calculator
An AI agent tool that calculates precise break-even metrics for cruise add-ons like drink, Wi-Fi, dining, and photo packages.
143m² Exclusive Renovation Assistant (renovation-143m)
An AI-powered, hyper-targeted renovation assistant specifically engineered for planning, budgeting, and visually simulating a 143m² 3-bedroom, 2-bathroom home layout.
UI Page Replication V1
An advanced frontend automation workflow designed to clone and replicate complex backend/admin UI pages with 100% fidelity using Playwright and AI layout analysis.
OpenClaw Traffic Guardian
OpenClaw Traffic Guardian is a baseline runtime security monitoring tool that provides opt-in HTTP/HTTPS proxy inspection, egress tracking, and policy review for AI agents.
InferSports Odds & Scores
A read-only, keyless CLI-driven AI agent skill providing real-time football and basketball scores, sharp betting lines, and de-vigged fair probabilities.
Featured*
