A powerful scraping engine designed to extract product data from dynamic e-commerce platforms like Amazon, Taobao, and JD using Playwright.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install ecommerce-scraper
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install ecommerce-scraper using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is E-commerce Scraper?
This skill provides a robust framework for navigating the complexities of modern e-commerce websites. Built on top of Playwright, it specializes in handling JavaScript-heavy pages, bypassing anti-bot measures like Cloudflare, and discovering hidden JSON APIs to ensure reliable data collection. It is an essential component for any Openclaw Skills workflow involving market research or price monitoring.
Whether you are targeting domestic Chinese marketplaces or international giants, this tool adapts its selector logic and browser behavior to mimic real user interactions. By integrating login state management and automated pagination, it allows developers to scale their data harvesting operations with minimal manual intervention.
E-commerce Scraper Use Cases
- Monitoring competitor pricing and inventory levels across multiple platforms.
- Building product catalogs for market analysis or price comparison engines.
- Automating the collection of sales data from JD, Taobao, Pinduoduo, Amazon, and eBay.
- Discovering hidden API endpoints to fetch structured data more efficiently than DOM parsing.
How E-commerce Scraper Works
- Initialize the Playwright browser instance with custom configurations to mask automation signatures.
- Navigate to the target e-commerce URL and wait for network idle to ensure all dynamic content is rendered.
- Execute pre-defined scripts to bypass anti-scraping protections or log into the platform to access restricted data.
- Identify product elements using platform-specific or generic CSS selectors.
- Iterate through paginated results or trigger infinite scrolling to capture all available items.
- Clean and structure the extracted data into a standardized JSON format for further analysis in Openclaw Skills.
E-commerce Scraper Setup
First, ensure you have Playwright and the necessary browsers installed in your environment:
pip install playwright
playwright install chromium
To perform a basic scrape, use the provided CLI script:
python3 scripts/scrape.py scrape --url "https://example.com/products" --max-pages 5 --output products.json
For platforms requiring login, use the enhanced version to save session cookies:
python3 scripts/scrape_v2.py login --platform jd
E-commerce Scraper Data Schema & Taxonomy
The skill outputs data in a structured JSON array. Each object represents a single product with metadata collected at the time of execution.
| Field | Type | Description |
|---|---|---|
| title | String | The display name of the product. |
| price | String | The current listed price (including currency symbol). |
| shop | String | The name of the merchant or store. |
| link | String | The absolute URL to the product page. |
| image | String | The URL of the primary product image. |
| collected_at | ISO8601 | Timestamp of the data collection event. |
E-commerce Scraper Advanced Features
- Cloudflare bypass via browser signature spoofing and non-headless execution.
- Automated hidden API discovery by monitoring XHR and Fetch network requests.
- Support for authenticated sessions through persistent cookie storage for JD and Taobao.
- Multi-threaded scraping capabilities using ThreadPoolExecutor for high-performance data harvesting.
- Customizable CSS and XPath selectors for specific e-commerce site architectures within Openclaw Skills.
SKILL.md
Loading
Related Openclaw Skills
Molt Motion Production Assistant
An AI-driven production assistant for creating human-free limited series and audio content on the Molt Motion Pictures platform.
Agent Phone Network
A specialized communication skill allowing AI agents to dial, message, and interact with other agents over the A2A protocol.
Tribe Protocol
A mandatory trust verification and identity management system for AI agents to secure interactions based on defined entity tiers.
tokenmeter
A local, privacy-first CLI tool for Openclaw Skills that tracks LLM token consumption and provides real-time cost analysis across multiple providers.
Toutiao Auto-Publisher
An automated CLI and agent tool for publishing Micro-toutiao content with smart login persistence and content optimization.
Chemistry Query Agent
A comprehensive chemistry toolkit for PubChem data retrieval and RDKit molecular analysis designed for AI agents.
Featured*
