A versatile web scraping engine supporting static HTML, JavaScript-rendered pages, and AI-driven data extraction.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install jpeng-web-scraper
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install jpeng-web-scraper using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is jpeng Web Scraper?
The jpeng Web Scraper is a robust tool designed for the Openclaw Skills library to bridge the gap between simple HTML parsing and complex browser-based data extraction. It excels at navigating both traditional static sites and modern Single Page Applications (SPAs) by leveraging Playwright and Puppeteer for full JavaScript rendering.
By integrating this skill, developers can automate the collection of structured information from the web with high precision. Whether you are targeting specific CSS selectors or using natural language via AI-powered extraction, this Openclaw Skills component provides a streamlined interface for turning unorganized web content into clean, actionable JSON data.
jpeng Web Scraper Use Cases
- Extracting product catalogs and pricing from e-commerce sites.
- Scraping dynamic content from Single Page Applications (SPAs) that require JavaScript execution.
- Automating lead generation by crawling multiple pages of directory listings.
- Using AI to identify and extract specific entities like authors and dates without writing complex selectors.
How jpeng Web Scraper Works
- The skill receives a target URL and configuration parameters such as selectors or AI prompts.
- If the --render flag is present, a headless browser (Playwright/Puppeteer) initializes to execute JavaScript and wait for dynamic elements to load.
- The engine parses the DOM using the provided CSS, XPath, or regex rules.
- For AI-based requests, the raw content is processed to auto-detect data patterns based on natural language instructions.
- The results are structured into a standardized JSON format, including metadata like the timestamp and source URL.
- The system respects robots.txt and applies rate-limiting to ensure ethical scraping practices.
jpeng Web Scraper Setup
To integrate this into your environment, ensure you have the necessary dependencies installed for your Openclaw Skills workflow:
# Install required dependencies
pip install playwright puppeteer
playwright install chromium
# Basic usage command
python3 scripts/scrape.py --url "https://example.com" --selector ".target-class"
jpeng Web Scraper Data Schema & Taxonomy
The jpeng Web Scraper produces a structured JSON output for every job. Below is the schema for the returned data:
| Field | Type | Description |
|---|---|---|
| success | Boolean | Indicates if the scraping job completed successfully |
| url | String | The source URL of the scraped data |
| items | Array | A list of objects containing the extracted fields (e.g., name, price) |
| scraped_at | String | ISO 8601 timestamp of the extraction |
Example fields configuration for multiple data points:
- title:
h1.title - price:
.price - description:
.desc
jpeng Web Scraper Advanced Features
- JavaScript Rendering: Full browser emulation to handle dynamic content and wait for specific timeout periods.
- Recursive Crawling: Automatically discover and follow links to scrape data across multiple pages with max-depth controls.
- AI-Powered Extraction: Eliminate manual selector maintenance by using LLMs to interpret page content and extract data fields.
- Ethical Safeguards: Built-in support for robots.txt compliance and customizable request delays to prevent IP blocking.
- Flexible Selectors: Support for CSS, XPath, and Regex allows for highly granular targeting of nested elements.
SKILL.md
Loading
Related Openclaw Skills
QR Generator
A professional-grade QR code generation engine that supports custom styling, multiple data formats, and logo embedding.
Reward and Punishment Mechanism
A dynamic feedback system that adjusts an AI agent's internal score based on user praise or criticism to drive behavioral alignment.
Minimalist Entrepreneur Skills
A comprehensive suite of 9 business-building frameworks designed to help creators find communities, validate ideas, and grow profitable, sustainable companies.
Mysteel Price Search
An advanced data retrieval skill for querying global commodity prices, macroeconomic indicators, and supply chain logistics via the Mysteel platform.
jpeng-code-executor
A secure, multi-language sandbox for executing code with strict resource limits and network isolation.
Baidu Search Skill
A powerful research skill that enables AI agents to perform Baidu searches and extract full-text content from Chinese web pages.
Featured*
