A set of architectural principles for coordinating stateless scraping and stateful Cloud Browser interactions using a connected Scrapfly MCP server.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install scrapfly-agent-rules
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install scrapfly-agent-rules using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is scrapfly-agent-rules?
The scrapfly-agent-rules guide outlines structured directives for autonomous AI agents integrated with the Scrapfly MCP server. By standardizing workflows across multiple scraping, screenshot, extraction, and browser-control tools, this skill helps developers build resilient automation pipelines.\n\nImplementing these guidelines within the Openclaw Skills ecosystem ensures that agents dynamically choose between lightweight stateless queries and continuous cloud-based browser sessions, minimizing execution costs while maintaining high execution success rates.
scrapfly-agent-rules Use Cases
- Deciding between stateless page fetching and stateful cloud browser automation to manage usage budgets.\n- Automatically bypassing anti-bot challenges and captchas through structured verification loops.\n- Coordinating multi-step browser interactions such as logging in, filling forms, and managing navigation states.\n- Safeguarding scraping operations from soft-block failures by applying adaptive request retries.
How scrapfly-agent-rules Works
- Determine the Session Lifecycle: The agent evaluates if the user's task requires interactive sessions; otherwise, it defaults to stateless scraping tools.\n2. Manage Browser State: When stateful interaction is required, it instantiates a Cloud Browser session, taking care to close the session once the task concludes.\n3. Identify and Overcome Blocks: Uses validation checks on returned page contents to detect soft-blocks, escalating queries to browser_unblock when challenged.\n4. Interact via Declared APIs: Prioritizes WebMCP tools exposed by the page over manual DOM-level clicks or keystrokes to ensure robustness against refactoring.\n5. Synchronize Page State: Executes page snapshot commands immediately after any state-changing mutation to ensure selector identifiers remain stable.\n6. Validate Metrics Prior to Alerting: Discovers baseline target metrics and previews evaluator states before creating system-level alerts.
scrapfly-agent-rules Setup
To apply these guidelines using Openclaw Skills, connect your agent to a local or remote Scrapfly MCP server. Configure your environment variables as shown below:\n\nbash\n# Configure local stdio MCP (reference agents default)\nexport SCRAPFLY_API_KEY=scp-live-your-key-here\n\n# Or point to a remote MCP endpoint\nexport SCRAPFLY_MCP_URL=https://mcp.scrapfly.io/mcp\nexport SCRAPFLY_API_KEY=scp-live-your-key-here\n\n\nNote that tool discovery is handled dynamically at connection time, and no heavy SDK installations are required for the rule evaluator itself.
scrapfly-agent-rules Data Schema & Taxonomy
| Item | Source | Type / Role | Description |\n| :--- | :--- | :--- | :--- |\n| SCRAPFLY_API_KEY | Environment | Authentication | Required to access Scrapfly API endpoints and MCP services |\n| SCRAPFLY_MCP_URL | Environment | Endpoint URL | Target host for remote Scrapfly Model Context Protocol servers |\n| tools/list | Dynamic | MCP Schema | Automatic discovery of available scraping and automation tools |\n| take_snapshot | Mutation | State Identifier | Captures stable page states and DOM elements for subsequent agent steps |
scrapfly-agent-rules Advanced Features
- Single-Turn Constraints: Prevents race conditions during browser state mutations by enforcing a single tool execution per step.\n- Dynamic ASP Escalation: Transitions seamlessly from standard requests to Active Scraping Protection bypasses based on soft-block signals.\n- WebMCP Native Tools: Targets declared APIs directly to maintain scrapers even after underlying DOM updates change.\n- Data-Grounded Alerting: Uses dry-run evaluation ticks on past metrics to calibrate alert systems, preventing false-positive noise.
SKILL.md
Loading
Related Openclaw Skills
Kubernetes Pod Logs Analyzer
An AI-powered log aggregation and anomaly detection tool for live monitoring of Kubernetes namespace pods.
Scrapfly Crawler
A Python-based crawler skill utilizing the Scrapfly Crawler API to automate site-wide data extraction, manage crawl depth, and bypass anti-bot mechanisms.
Scrapfly Screenshot
A Python-based integration utilizing the Scrapfly API to capture high-quality, Javascript-rendered, and geo-targeted screenshots of any web page.
Scrapfly Scraper
An advanced web scraper leveraging the Scrapfly API to extract web data with anti-bot bypass, proxy rotation, and headless browser rendering.
Ghost Eye
Ghost Eye functions as an image preprocessing bridge, converting visual data into rich OCR text and scene descriptions for pure-text LLMs.
scrapfly-cli
A comprehensive command-line interface and MCP server to orchestrate Scrapfly scraping, automated rendering, screenshots, and remote CDP browser sessions.
Featured*
