An advanced web scraping SDK for Go that enables seamless proxy rotation, anti-bot bypass, JavaScript rendering, and automated data extraction.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install scrapfly-sdk-go
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install scrapfly-sdk-go using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Scrapfly Go SDK?
The Scrapfly Go SDK is a powerful development tool designed to streamline web scraping workflows within Go applications. By implementing the github.com/scrapfly/go-scrapfly module, developers can bypass restrictive anti-bot solutions such as Cloudflare, DataDome, PerimeterX, and Kasada using Anti-Bot Protection (ASP).
This skill provides complete access to headless browser capabilities, rendering JavaScript-heavy targets, capturing screenshots, and outputting data in highly clean, LLM-friendly structural formats like Markdown. Additionally, it integrates built-in goquery selector handling to rapidly parse HTML contents within native Go code codebases.
Scrapfly Go SDK Use Cases
- Scraping raw HTML, JSON, text, or markdown directly from any web target into a Go codebase.
- Circumventing aggressive anti-bot and DDoS protection walls automatically.
- Extracting dynamic data from modern single-page applications (SPAs) that require complete JavaScript execution.
- Generating automated visual snapshots of targeted web interfaces via screenshots or running natural language AI extraction prompts.
How Scrapfly Go SDK Works
- Initialize the scraping engine client by supplying a authenticated Scrapfly API key configuration variable.
- Construct a customized
ScrapeConfigobject defining target parameters such as the target URL, proxy pool preference, and regional geo-targeting. - Execute the request through the client, triggering Scrapfly's backend cloud browsers to process cookies, render scripts, or clear anti-bot mechanisms.
- Deliver an optimized API response object back to the Go application environment for instant schema extraction, saving, or structural querying.
Scrapfly Go SDK Setup
Install the required module dependency into your project workspace using the Go toolchain, and set up your authenticated API key credential:
go get github.com/scrapfly/go-scrapfly
export SCRAPFLY_API_KEY="your-api-key"
Scrapfly Go SDK Data Schema & Taxonomy
The SDK exposes a structured configuration layout and yields responses containing rich body metadata:
| Configuration Field | Data Type | Key Description |
|---|---|---|
URL |
string | Target web endpoint to fetch |
ASP |
boolean | Activates Anti-Bot Protection bypass logic |
ProxyPool |
string | Chooses proxy network type (e.g. residential vs data-center) |
RenderJS |
boolean | Toggles headless browser execution |
Format |
string | Formats output as raw, clean_html, json, markdown, or text |
Responses can be handled directly via apiResponse.Result.Content or processed utilizing a built-in goquery layout document instance by running apiResponse.Selector().
Scrapfly Go SDK Advanced Features
- Smart proxy integration leveraging specialized residential or data-center pools tailored for higher success rates.
- Granular render management allowing execution delays via
RenderingWaitand conditional loading triggers usingWaitForSelector. - Full multi-agent automation capabilities with direct extraction configs using native LLM prompts (
ExtractionPrompt) across scraped HTML payloads.
SKILL.md
Loading
Related Openclaw Skills
Scrapfly Cloud Browser
Automate cloud browsers using the Scrapfly Cloud Browser API with Python Playwright for residential proxy rotation and anti-bot fingerprint management.
StablePay Agent Bootstrap
A dedicated setup skill to bootstrap, validate, and integrate the StablePay MCP server and CLI utility for AI coding agents.
kb_web_query
An intelligent Q&A skill that leverages team overview documents to guide and ground external web search queries.
X (Twitter) Auto-Posting & Content Operations
An automated X (Twitter) content operations workflow that takes target keywords from research to published, tracked posts within a real browser session.
Scrapfly Alerting
A threshold-based monitoring integration to watch Scrapfly account metrics and trigger notifications via webhooks, email, or in-app alerts.
Scrapfly TypeScript SDK
A TypeScript and JavaScript SDK for Scrapfly designed to handle complex web scraping, JavaScript rendering, anti-bot bypass, and AI-driven data extraction.
Featured*
