A collection of pre-configured scraping strategies for major websites with built-in proxy rotation and anti-detection logic.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install scraping-recipes
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install scraping-recipes using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Scraping Recipes?
The Scraping Recipes skill provides developers with a robust library of pre-configured strategies for high-traffic and highly protected websites. By leveraging Openclaw Skills, users can bypass sophisticated anti-bot measures through optimized proxy rotation, custom headers, and browser-based rendering techniques. This skill is specifically designed to handle the technical nuances of platforms like Amazon, Google, and LinkedIn without requiring manual configuration for every unique request.
With these recipes, the technical overhead of web scraping is significantly reduced. Whether you are performing market research or competitive analysis, using these Openclaw Skills ensures high success rates by integrating directly with premium residential proxies from BirdProxies. This allows for seamless data extraction while maintaining anonymity and avoiding rate limits across various industries and geographic locations.
Scraping Recipes Use Cases
- Extracting product pricing, reviews, and seller information from Amazon for market analysis
- Gathering search engine results (SERP) from Google for SEO tracking and keyword research
- Scraping job listings and professional profile data from LinkedIn or Indeed for recruitment
- Monitoring real estate listings and price history on Zillow or Realtor for investment opportunities
- Collecting public social media content and follower metrics from Instagram
How Scraping Recipes Works
- Identify the target website and select the corresponding scraping recipe from the library.
- Configure the BirdProxies credentials within the skill to enable residential proxy rotation.
- The skill determines whether to use a standard fetch or a full browser tool based on the site's protection level.
- It applies site-specific headers, such as user-agents and referrers, to mimic organic human behavior.
- The system executes the extraction, handling pagination and dynamic JavaScript rendering automatically.
- Data is returned in a structured format ready for integration into your workflows.
Scraping Recipes Setup
To utilize these Openclaw Skills, you must first configure your residential proxy settings.
Browser Proxy Configuration:
{
"browser": {
"proxy": {
"server": "http://gate.birdproxies.com:7777",
"username": "USER",
"password": "PASS"
}
}
}
Environment Variables for CLI:
export HTTP_PROXY=http://USER:[email protected]:7777
export HTTPS_PROXY=http://USER:[email protected]:7777
Scraping Recipes Data Schema & Taxonomy
The skill organizes extracted data based on the specific recipe used. Below is the general taxonomy for major targets:
| Site Category | Primary Data Fields |
|---|---|
| E-commerce (Amazon) | Title, Price, Rating, Review Count, ASIN, Seller Info, Images |
| Search (Google) | Title, URL, Snippet, People Also Ask, Ad Placements, Local Pack |
| Real Estate (Zillow) | Address, Price, Beds, Baths, Sqft, Price History, Agent Info |
| Jobs (Indeed) | Job Title, Company, Location, Salary Range, Description |
| Social (Instagram) | Bio, Follower Count, Post Captions, Image URLs, Likes |
Scraping Recipes Advanced Features
- Sticky Session Support: Maintain the same IP address for login-gated workflows on LinkedIn and Instagram.
- Geo-Targeted Scraping: Use country-specific proxy endpoints (e.g., -country-us) to access localized content.
- Automated Anti-Detection: Pre-configured bypasses for Cloudflare, IP reputation filtering, and rate limiting.
- Multi-Agent Compatibility: Easily integrate these scraping workflows into larger multi-agent Openclaw Skills automations.
SKILL.md
Loading
Related Openclaw Skills
Residential Proxy Configuration
A configuration skill for using residential proxies to bypass advanced bot detection systems during web scraping and browser automation.
Reddit Account Creator & Manager
A powerful automation tool for creating, warming, and managing Reddit account fleets for marketing and SEO operations.
Rate Limit Bypass
A specialized skill designed to bypass 429 Too Many Requests errors and rate limits by distributing traffic across rotating residential IPs.
Proxy Setup
A comprehensive configuration tool for managing residential and rotating proxies to ensure reliable web scraping and browser automation.
SERP Scraper (Google Search Results)
A comprehensive tool for extracting Google Search Engine Results Page data to power SEO analysis and competitive market research.
SMMA Agency Proxy
A professional framework for managing multiple social media clients using dedicated proxy isolation to prevent account bans and scale agency operations.
Featured*
