A comprehensive command-line interface and MCP server to orchestrate Scrapfly scraping, automated rendering, screenshots, and remote CDP browser sessions.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install scrapfly-cli
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install scrapfly-cli using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is scrapfly-cli?
The scrapfly-cli provides unified command-line control over Scrapfly's web scraping, screenshot generation, AI extraction, recursive crawling, and cloud-hosted browser execution capabilities. Built in Go, it outputs structured JSON on stdout, serving as an optimal utility for automated workflows and terminal integrations.
When incorporated as part of your Openclaw Skills, this CLI allows AI agents and local developers to route requests through proxy pools, bypass anti-bot mechanisms, retrieve fully rendered web contents, and manage long-running browser environments programmatically.
scrapfly-cli Use Cases
- Scraping complex JavaScript-rendered web pages while bypassing advanced anti-bot protections dynamically.
- Extracting clean structured data (markdown, text, or clean HTML) from target URLs using prompt-driven AI extraction models.
- Executing automated browser interactions like clicking, filling forms, and navigating across persistent multi-call daemon sessions.
- Capturing high-resolution visual screenshots of web layouts or specific DOM elements via standard CLI commands.
- Crawling websites recursively, collecting and parsing HAR or WARC archives for offline indexing and analysis.
How scrapfly-cli Works
- The user configures authentication via the command-line interface, local settings file, or environment variables.
- The CLI translates command flags (such as proxies, waiting selectors, and render options) into Scrapfly API payloads.
- Scrapfly's cloud browser infrastructure handles complex actions, solves captchas, and returns the requested data payload.
- The scrapfly-cli processes the response and emits a standardized JSON envelope containing structural metadata and the data payload.
scrapfly-cli Setup
Prerequisites
Install the latest binary from source using Go:
go install github.com/scrapfly/scrapfly-cli/cmd/scrapfly@latest
Alternatively, download a pre-built binary for your platform from the project releases page.
Authentication
Set your Scrapfly API key as an environment variable:
export SCRAPFLY_API_KEY=scp-live-...
Or persist it locally through the configuration tool:
scrapfly config set-key scp-live-...
MCP Server Integration
To expose scrapfly-cli as a toolset within your Openclaw Skills configuration, register the MCP server in your client environment configuration (e.g., Claude Desktop, Cursor, or Claude Code):
{
"mcpServers": {
"scrapfly": {
"command": "scrapfly",
"args": ["mcp", "serve"],
"env": {
"SCRAPFLY_API_KEY": "scp-live-..."
}
}
}
}
scrapfly-cli Data Schema & Taxonomy
Every command execution outputs a consistent JSON structured response schema, facilitating easy parsing in terminal-based pipelines.
Standard Output Envelope
| Field | Type | Description |
|---|---|---|
success |
boolean | Confirms if the operation succeeded or failed |
product |
string | Identifies the service product (e.g., scrape, screenshot, extraction, crawl) |
data |
object | Contains the actual result payload when success is true |
error |
object | Contains the detailed error model with code, message, and http_status on failure |
scrapfly-cli Advanced Features
- Persistent Browser Sessions: Keep a CDP session open using a background daemon, preserving tabs, cookies, and AXTree references for sequential execution.
- Autonomous LLM Agents: Run localized loops with the
scrapfly agentcommand, letting integrated AI models choose actions dynamically to satisfy a text-based task instruction. - Bring-Your-Own-Proxy (BYOP): Run an exit-peer connector that establishes an mTLS tunnel to route Scrapfly browser traffic through your custom network egress.
- Crawl Parsing Utility: Direct inspection of crawling artifacts (WARC and HAR files) using local CLI parser commands without external analysis software.
- Alerting & Schedulers: Configure metrics rules for operational alerts and schedule recurring scraping tasks to land via configured webhooks.
SKILL.md
Loading
Related Openclaw Skills
Ghost Eye
Ghost Eye functions as an image preprocessing bridge, converting visual data into rich OCR text and scene descriptions for pure-text LLMs.
scrapfly-agent-rules
A set of architectural principles for coordinating stateless scraping and stateful Cloud Browser interactions using a connected Scrapfly MCP server.
Kubernetes Pod Logs Analyzer
An AI-powered log aggregation and anomaly detection tool for live monitoring of Kubernetes namespace pods.
Scrapfly Crawler
A Python-based crawler skill utilizing the Scrapfly Crawler API to automate site-wide data extraction, manage crawl depth, and bypass anti-bot mechanisms.
e2e-delivery
A continuous delivery orchestration skill that automates PingCode work item resolution, branching, merge requests, code reviews, and reporting.
File-Count Threshold Probe
A diagnostic utility that binary-searches multipart part-count limits to identify upload thresholds and prevent HTTP 413 errors.
Featured*
