A smart scraping tool that uses Playwright and curl to extract content from both static and dynamic JavaScript-heavy novel websites.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install novel-scraper-spa
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install novel-scraper-spa using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Novel Scraper SPA?
Novel Scraper SPA is a robust utility designed for the Openclaw Skills ecosystem to solve the challenge of extracting text from modern web architectures. It features an intelligent detection engine that identifies whether a target website is a static page or a Single Page Application (SPA). By leveraging Playwright for JavaScript rendering and curl for static content, it provides a high-performance solution for gathering digital content from diverse web sources while maintaining a developer-friendly interface.
Novel Scraper SPA Use Cases
- Extracting novel chapters from dynamic websites that require JavaScript execution.
- High-speed scraping of static novel repositories using optimized curl requests.
- Automating the collection of web-based reading material into a local workspace for offline use.
- Bypassing rendering hurdles on complex SPA architectures like Biquge.
How Novel Scraper SPA Works
- The user provides a target URL, book name, and chapter number via the CLI.
- The skill analyzes the target site to determine its architectural type (Static vs. SPA).
- For static sites, it uses curl to quickly pull the source; for SPAs, it launches a Playwright-controlled browser to render the page content.
- It parses the resulting content to isolate the book title and specific chapter text.
- The processed content is automatically saved into the organized Openclaw Skills workspace directory.
Novel Scraper SPA Setup
To integrate this tool into your Openclaw Skills workflow, ensure you have the necessary environment and dependencies ready.
# Install the Playwright browser engine
pip install playwright
playwright install
# Run the scraper
python3 scripts/scraper.py --url <URL> --book <BookName> --chapter <Number>
Novel Scraper SPA Data Schema & Taxonomy
The skill manages extracted data using a structured directory approach within the Openclaw Skills environment:
| Component | Path / Details |
|---|---|
| Storage Root | ~/.openclaw/workspace/novels/ |
| File Structure | Content is organized by <book_name>/<chapter>.txt |
| Rendering Logs | Accessible via the -v flag for debugging parsing logic |
Novel Scraper SPA Advanced Features
- Intelligent engine switching between high-speed curl and full browser rendering.
- Force SPA mode (
--force-spa) to ensure content is captured even from heavily obfuscated sites. - Verbose logging capabilities for deep inspection of the scraping lifecycle.
- Automated workspace management within the Openclaw Skills directory structure.
SKILL.md
Loading
Related Openclaw Skills
Crypto Market Rank
A comprehensive crypto analytics skill for tracking token rankings, smart money movements, and trader performance across major blockchains.
Binance P2P Trading Assistant
A specialized tool for querying Binance P2P market data and personal order history using natural language queries.
Binance Fiat
A powerful tool for managing fiat currency transactions and history on Binance through authenticated API requests.
Auto Deep Research
An advanced research skill that autonomously decomposes queries, performs parallel searches, and verifies information across multiple independent sources.
Xiaohongshu Full Auto
A comprehensive automation pipeline that manages the entire Xiaohongshu content lifecycle from trend discovery to final publication.
WeCom MCP Configuration Guide
A technical guide to integrating Enterprise WeChat (WeCom) with AI agents for automated messaging and task management.
Featured*
