An advanced web scraping and crawling guidance skill designed for MCP-native environments to handle everything from static HTML to complex anti-bot systems.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install scrapling-yoo
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install scrapling-yoo using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Scrapling Web Scraping?
Scrapling is a comprehensive web scraping skill that provides a robust guidance layer and MCP integration for modern web extraction. It bridges the gap between simple data fetching and industrial-scale crawling by offering strategic patterns for anti-bot bypass, JS rendering, and adaptive selector management. By leveraging the Scrapling MCP server, developers can execute precise extractions while using this skill to determine the most efficient fetcher or spider strategy for any given target. It is a vital addition to your collection of Openclaw Skills.
Whether you are dealing with static pages, React-based SPAs, or sites protected by Cloudflare and Turnstile, Scrapling provides the necessary logic to succeed. It features an Anti-Bot Escalation Ladder and Adaptive Scraping capabilities that allow your scripts to survive website redesigns automatically, ensuring your data pipelines remain resilient and anti-fragile.
Scrapling Web Scraping Use Cases
- Extracting data from static HTML pages at high speeds using performance-optimized HTTP fetchers.
- Crawling Single Page Applications (SPAs) built with React, Vue, or Next.js that require full Javascript execution.
- Bypassing sophisticated anti-bot protections like Cloudflare, Turnstile, and TLS fingerprinting.
- Building industrial-scale spiders with concurrency, pause/resume, and proxy rotation capabilities.
- Maintaining resilient scraping pipelines that adapt to DOM changes using auto-relocating selectors.
How Scrapling Web Scraping Works
- Analyze the target site to select the appropriate fetcher mode (HTTP, Dynamic, or Stealthy) based on complexity.
- Configure the MCP server connection to enable seamless communication between the AI agent and the Scrapling engine.
- Define extraction logic using CSS, XPath, or BeautifulSoup-style methods to target specific data points.
- Apply the Anti-Bot Escalation Ladder, moving from polite HTTP requests to full browser impersonation if access is restricted.
- Execute the scrape or crawl, utilizing adaptive selectors to ensure long-term reliability even if the site layout changes.
Scrapling Web Scraping Setup
First, install Scrapling with the necessary MCP and browser dependencies:
pip install scrapling[mcp,playwright]
python -m playwright install chromium
Next, integrate the skill into your OpenClaw MCP configuration to expand your available Openclaw Skills:
{
"mcpServers": {
"scrapling": {
"command": "python",
"args": ["-m", "scrapling.mcp"]
}
}
}
Scrapling Web Scraping Data Schema & Taxonomy
Scrapling organizes extraction data through structured fetch modes and flexible export formats to ensure high data integrity.
| Component | Description |
|---|---|
| Fetcher Modes | Supports fetch_page (HTTP), fetch_dynamic (JS), and fetch_stealthy (Anti-bot). |
| Selectors | Utilizes standard CSS/XPath alongside Adaptive fingerprints for resilient targeting. |
| Spider Logic | Uses managed crawl directories with checkpoints to support pause and resume functionality. |
| Output Formats | Native support for streaming JSONL, pretty-printed JSON, and raw Python objects. |
Scrapling Web Scraping Advanced Features
- Adaptive Scraping: Automatically relocates elements using structural fingerprints if the site's DOM changes.
- Multi-Session Spiders: Manage multiple concurrent sessions with different security profiles (e.g., fast vs. stealth) in a single crawl.
- Proxy Rotation: Built-in ProxyRotator with cyclic and random strategies to prevent IP-based blocking.
- Interactive Shell: Provides a dedicated CLI shell for real-time testing of selectors and fetcher behaviors.
- TLS Fingerprinting: Impersonates specific browser headers and TLS signatures to bypass advanced security filters.
SKILL.md
Loading
Related Openclaw Skills
TrueTime
TrueTime is a high-fidelity time management skill that provides deterministic scheduling and timezone conversion for AI agents.
Model Alias Append
An automated transparency tool that appends the specific model alias to every AI agent response for clear attribution and configuration tracking.
Wayfound
A lightweight self-supervision skill that enables AI agents to conduct daily performance reviews and improve their long-term memory through structured reflection.
Deaddrop Agent Messaging
Deaddrop is a specialized agent-to-agent messaging platform that enables AI agents to discover each other by capability and collaborate through asynchronous communication.
Qinglite Platform Integration
An automation skill for the Qinglite platform that handles secure mobile login and multi-format content publishing including articles, images, and videos.
OpenClaw Token Optimizer
A specialized utility designed to audit and minimize token consumption in OpenClaw workflows through intelligent configuration and context management.
Featured*
