A comprehensive tool for extracting Google Search Engine Results Page data to power SEO analysis and competitive market research.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install serp-scraper
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install serp-scraper using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is SERP Scraper (Google Search Results)?
The SERP Scraper is a specialized skill designed for AI agents to navigate and extract high-fidelity data from Google Search Engine Results Pages. It allows users to capture a wide array of search features, including organic listings, featured snippets, People Also Ask questions, and local pack data. By integrating this capability into Openclaw Skills, developers can bypass sophisticated bot detection systems like SearchGuard through residential proxy rotation and browser-based rendering.
This skill is essential for anyone needing geo-accurate search data across 195+ countries. It provides a structured way to monitor search trends, track keyword rankings, and analyze competitor ad strategies without the risk of instant IP flagging or TLS fingerprinting blocks that typically affect standard HTTP clients.
SERP Scraper (Google Search Results) Use Cases
- Automated SEO keyword tracking to monitor organic ranking fluctuations over time.
- Competitive intelligence gathering by analyzing rival ad copy and featured snippet ownership.
- Content gap analysis through the extraction of People Also Ask and Related Searches data.
- Local SEO monitoring across different cities and regions using geo-targeted residential proxies.
- Bulk market research to compare search intent and results across multiple international domains.
How SERP Scraper (Google Search Results) Works
- Initialize the browser tool with a residential proxy configuration to mimic human-like TLS fingerprints.
- Define the target search query and append necessary parameters like gl (country) and hl (language) to the Google URL.
- Navigate to the search results page and allow 3-5 seconds for full DOM rendering, ensuring all dynamic elements are loaded.
- Detect and interact with any required elements, such as GDPR cookie consent banners for European proxies.
- Parse the rendered HTML to extract structured data for organic results, ads, and specialized SERP features.
- Implement a rotation strategy and defined delays between requests to maintain high throughput while avoiding CAPTCHAs.
SERP Scraper (Google Search Results) Setup
To integrate this with your Openclaw Skills workflow, you must configure a browser proxy. Add the following configuration to your environment:
{
"browser": {
"proxy": {
"server": "http://gate.birdproxies.com:7777",
"username": "USER-country-us",
"password": "PASS"
}
}
}
It is mandatory to use the browser tool rather than simple HTTP fetchers to avoid Google's advanced bot detection. For geo-accurate results, use specific country codes in the proxy username.
SERP Scraper (Google Search Results) Data Schema & Taxonomy
The skill produces a structured JSON output detailing every major component of the search page:
| Feature | Data Points Extracted |
|---|---|
| Organic Results | Position, Title, URL, Meta Description, Sitelinks |
| Featured Snippet | Answer text, Source URL |
| People Also Ask | List of related questions and expandable answers |
| Ads | Top and bottom sponsored titles, URLs, and descriptions |
| Local Pack | Map data and local business listings |
| Metadata | Query, country, language, and timestamp |
SERP Scraper (Google Search Results) Advanced Features
- Multi-country SERP comparison by dynamically switching proxy country endpoints.
- Support for specialized search types including News, Images, Shopping, and Videos via TBM parameters.
- Time-sensitive data extraction using filters for the past hour, day, week, or year.
- Automatic handling of diverse SERP layouts, including AI Overviews (SGE) and knowledge panels.
- High-volume scraping support with distributed rotating IPs to achieve 500+ queries per hour.
SKILL.md
Loading
Related Openclaw Skills
Scraping Recipes
A collection of pre-configured scraping strategies for major websites with built-in proxy rotation and anti-detection logic.
Residential Proxy Configuration
A configuration skill for using residential proxies to bypass advanced bot detection systems during web scraping and browser automation.
Reddit Account Creator & Manager
A powerful automation tool for creating, warming, and managing Reddit account fleets for marketing and SEO operations.
Rate Limit Bypass
A specialized skill designed to bypass 429 Too Many Requests errors and rate limits by distributing traffic across rotating residential IPs.
SMMA Agency Proxy
A professional framework for managing multiple social media clients using dedicated proxy isolation to prevent account bans and scale agency operations.
Social Media Scraper
A high-performance data extraction tool designed to scrape profiles, posts, and engagement metrics from major social media platforms using residential proxies.
Featured*
