An advanced web scraper leveraging the Scrapfly API to extract web data with anti-bot bypass, proxy rotation, and headless browser rendering.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install scrapfly-scraper
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install scrapfly-scraper using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Scrapfly Scraper?
The Scrapfly Scraper integration enables headless browser automation and web scraping through the Scrapfly API. Designed for developers and automated workflows, it handles web harvesting challenges such as bypassing anti-bot solutions, executing client-side JavaScript, and pulling structured data.
This integration is designed to work with Openclaw Skills, allowing agents to ingest clean markdown or raw HTML from dynamically loaded pages while rotating residential proxies to bypass geo-blocks.
Scrapfly Scraper Use Cases
- Extracting web page data formatted as clean markdown, JSON, or raw HTML.
- Bypassing modern anti-bot protections including Cloudflare, DataDome, PerimeterX, and Kasada.
- Collecting regional data through rotating residential proxies with geo-targeting capabilities.
- Capturing browser XHR call data and screenshots from dynamic web pages.
- Persisting login states and user sessions across sequential scraping requests.
How Scrapfly Scraper Works
- Initialization: The ScrapflyClient is initialized using a Scrapfly API key retrieved from system environment variables.
- Configuration: A ScrapeConfig object defines the targeted URL, HTTP method, headers, cookies, and proxy requirements.
- Execution: The client executes the scrape request. If anti-bot security is met, the Anti Scraping Protection (ASP) engine actively works to circumvent blockers.
- Headless Browser Execution: If JavaScript rendering is enabled, a cloud browser renders the page, executes custom JS scenarios, and captures visual screenshots or dynamically generated nodes.
- Response Processing: A ScrapeApiResponse object yields status codes, headers, and the processed text, HTML, or markdown content.
Scrapfly Scraper Setup
To set up the Scrapfly SDK, install the package using pip:
pip install scrapfly-sdk
Set your API key as an environment variable in your terminal configuration:
export SCRAPFLY_API_KEY="your_scrapfly_api_key_here"
Scrapfly Scraper Data Schema & Taxonomy
The integration utilizes configuration parameters to structure requests and produces structured response objects.
ScrapeConfig Core Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
url |
str | Required | Target web page address. |
asp |
bool | False |
Bypasses Anti Scraping Protection when enabled. |
render_js |
bool | False |
Renders client-side JavaScript via headless browser. |
format |
str | "raw" |
Data output formats: "raw", "clean_html", "json", "markdown", or "text". |
session |
str | None |
Maintains persistent cookie state and sticky proxy IPs. |
js_scenario |
list | None |
Pre-defined browser automation steps. |
ScrapeApiResponse Structure
| Attribute | Type | Description |
|---|---|---|
content |
str | The extracted page content in the requested format. |
scrape_result |
dict | Complete raw API metadata payload. |
status_code |
int | The status code returned by the Scrapfly API. |
selector |
Parsel.Selector | Built-in interface for running XPath and CSS queries on the retrieved document. |
Scrapfly Scraper Advanced Features
- JavaScript Scenarios: Define chronological browser commands (click, fill, wait_for_navigation, scroll) to navigate authentication flows and hidden interfaces.
- Concurrent Operations: Run high-volume scraping loops with automated rate-limiting management via concurrent scrape interfaces.
- State Retention: Pin session contexts with sticky proxy pools to emulate real user behavior across sequences of requests.
- Markdown Extraction: Output raw page HTML directly as structured markdown optimized for downstream processing by large language models.
SKILL.md
Loading
Related Openclaw Skills
Scrapfly Extraction
An integration for extracting structured data from raw web content, text, or markdown using the Scrapfly Extraction API and AI-driven models.
Scrapfly TypeScript SDK
A TypeScript and JavaScript SDK for Scrapfly designed to handle complex web scraping, JavaScript rendering, anti-bot bypass, and AI-driven data extraction.
Scrapfly Alerting
A threshold-based monitoring integration to watch Scrapfly account metrics and trigger notifications via webhooks, email, or in-app alerts.
Scrapfly Go SDK
An advanced web scraping SDK for Go that enables seamless proxy rotation, anti-bot bypass, JavaScript rendering, and automated data extraction.
Scrapfly Screenshot
A Python-based integration utilizing the Scrapfly API to capture high-quality, Javascript-rendered, and geo-targeted screenshots of any web page.
Scrapfly Crawler
A Python-based crawler skill utilizing the Scrapfly Crawler API to automate site-wide data extraction, manage crawl depth, and bypass anti-bot mechanisms.
Featured*
