A high-performance search and scraping skill that provides AI agents with structured SERP data and bot-resistant web content extraction.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install brightdata
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install brightdata using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Bright Data Web Search & Scraping?
The Bright Data skill is a robust integration designed for AI agents that require reliable access to live web data. By leveraging Bright Data's enterprise-grade APIs, this skill enables agents to perform deep Google searches and extract clean, readable markdown from any public URL. It is a critical component for Openclaw Skills that need to bypass CAPTCHAs and bot detection mechanisms while maintaining high data integrity.
This skill is particularly effective for research-intensive tasks where an agent must first discover relevant links and then digest their full content. It bridges the gap between a LLM's static training data and the real-time information available on the open web, providing a seamless bridge for automated data retrieval.
Bright Data Web Search & Scraping Use Cases
- Performing targeted market research or competitive analysis by querying Google SERP.
- Extracting full-text documentation or articles from websites to serve as context for LLM reasoning.
- Automating the discovery of specific file types or domain-specific information using advanced search operators.
- Bypassing sophisticated anti-bot protections on websites that block standard scraping tools.
How Bright Data Web Search & Scraping Works
- The AI agent initiates a search request using a specific query to find relevant URLs.
- The search tool returns a structured JSON object containing titles, descriptions, and links from Google.
- If the initial search results are insufficient, the agent can paginate through additional results by incrementing the cursor.
- Once a target URL is identified, the agent calls the scraping tool to fetch the raw HTML.
- The skill processes the HTML and returns a clean markdown version of the page, removing noise and CAPTCHAs automatically.
Bright Data Web Search & Scraping Setup
To deploy this skill within your Openclaw Skills environment, ensure you have a Bright Data account with active SERP and Web Unlocker zones.
# Set your Bright Data credentials
export BRIGHTDATA_API_KEY="your_api_key_here"
export BRIGHTDATA_SERP_ZONE="your_serp_zone_id"
export BRIGHTDATA_UNLOCKER_ZONE="your_unlocker_zone_id"
# Ensure your environment has the necessary binaries
# Required: bash, curl, sed
Bright Data Web Search & Scraping Data Schema & Taxonomy
The skill organizes data into structured search results and unstructured content blocks:
| Component | Type | Description |
|---|---|---|
| Search Result | JSON | An object containing an organic array with link, title, and description keys. |
| Scrape Result | String | A clean Markdown representation of the target URL's content. |
| Cursor | Integer | A 0-indexed page marker for paginating through Google Search results. |
Each search page typically yields approximately 10 organic results, allowing for precise control over data ingestion volume.
Bright Data Web Search & Scraping Advanced Features
- Intelligent pagination support to navigate deep into search engine results.
- Built-in bot detection and CAPTCHA bypass via the Bright Data Unlocker infrastructure.
- Automatic conversion of complex HTML structures into LLM-friendly clean markdown.
- Support for advanced Google search operators (e.g., site:, filetype:) to refine discovery workflows.
SKILL.md
Loading
Related Openclaw Skills
Refundable Hotel Search Skill
A specialized AI agent skill designed to find hotels with free cancellation policies and flexible booking options using the flyai CLI.
SYSU Anything Apple
A specialized macOS bridge that synchronizes Sun Yat-sen University (SYSU) academic and campus activities directly with Apple Calendar and Reminders.
Spoken Script Polish
Spoken Script Polish transforms rigid written text into natural, high-performance oral scripts for video content creators.
Speech Structure Coach
A professional coaching skill designed to transform presentations into high-impact speeches using the proven heuristics of Patrick Winston.
Rafting Spots
A high-performance CLI skill for AI agents to find and book whitewater rafting, river tubing, and water adventure experiences in real-time.
Runbook Generator
An automated tool that scans project infrastructure files to generate structured, step-by-step operational runbooks for deployment and maintenance.
Featured*
