ClearWeb provides AI agents with unrestricted, unblocked access to the entire public web through an integrated Bright Data CLI.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install clearweb
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install clearweb using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is ClearWeb?
ClearWeb is a powerful bridge between AI agents and the live web, replacing fragile native tools like web_fetch or curl with the robust Bright Data infrastructure. By utilizing Openclaw Skills integrated with the bdata CLI, agents can bypass CAPTCHAs, handle complex JavaScript rendering, and rotate proxies automatically to ensure 100% reliable data retrieval.
This skill is designed for terminal-based AI agents that need to perform real-world tasks such as monitoring competitors, gathering leads, or reading documentation from sites that typically employ heavy anti-bot measures. It transforms the web into a queryable database, providing clean markdown or structured JSON instead of raw, noisy HTML.
ClearWeb Use Cases
- Market Research: Search across Google, Bing, and Yandex to gather competitive intelligence and price monitoring data.
- Lead Generation: Extract structured professional data directly from platforms like LinkedIn, Amazon, and Crunchbase.
- Content Aggregation: Scrape news articles, documentation, or social media feeds while bypassing paywalls and anti-scraping blocks.
- Social Media Monitoring: Track engagement and sentiment by pulling comments and profile data from Instagram, TikTok, and YouTube.
- Technical Documentation: Read and synthesize information from JS-heavy documentation sites like Docusaurus or GitBook.
How ClearWeb Works
- Request Initialization: The AI agent identifies a need for external information (search, scrape, or specific data extraction).
- Command Selection: Based on a built-in decision tree, the agent selects between
bdata searchfor discovery,bdata pipelinesfor known platforms, orbdata scrapefor general URLs. - Automated Bypassing: The Bright Data network handles proxy rotation, geo-targeting, and CAPTCHA solving in the cloud.
- Data Normalization: The tool processes the web content and returns it in a format optimized for LLMs, such as clean Markdown or structured JSON.
- Synthesization: The AI agent receives the clean data to perform analysis, comparison, or reporting.
ClearWeb Setup
First, ensure the Bright Data CLI is installed on your system:
# Install via curl for macOS / Linux
curl -fsSL https://cli.brightdata.com/install.sh | bash
# Or install via npm
npm install -g @brightdata/cli
Next, perform a one-time authentication to link your account and auto-configure proxy zones:
bdata login
Verify that your configuration is active:
bdata config
ClearWeb Data Schema & Taxonomy
ClearWeb organizes data into three primary output types to ensure compatibility with different agent workflows:
| Data Type | Description | Format |
|---|---|---|
| SERP Data | Organic results, ads, and 'People Also Ask' from search engines. | JSON |
| Scraped Content | Cleaned webpage text stripped of boilerplates. | Markdown / HTML |
| Pipeline Objects | Typed, platform-specific schema (e.g., Amazon Product, LinkedIn Profile). | JSON / CSV |
| Visual Evidence | Full-page screenshots for visual verification. | PNG |
Metadata such as job status, job IDs for async tasks, and account budget remaining can be queried via the bdata status and bdata budget commands.
ClearWeb Advanced Features
- Geo-Targeting: Access the web as if you were in a specific country using the
--countryflag. - Asynchronous Processing: Handle heavy scraping tasks or large batches without blocking the agent's primary loop using
--async. - Pre-built Extractors: Access over 40 specialized pipelines for major social and e-commerce platforms.
- Pipe-friendly Architecture: Seamlessly integrate with other CLI tools like
jqorxargsfor complex data processing workflows. - Automated Zone Management: Automatically creates and manages
cli_unlockerandcli_browserzones upon login.
SKILL.md
Loading
Related Openclaw Skills
YouTube Summary Skill
An automated utility for extracting YouTube transcripts and generating structured, multi-lingual summaries directly within your AI agent workflow.
Context Loader
An advanced context loading skill that retrieves business knowledge networks, object instances, and subgraph relationships for AI agents.
PDF Studio
A professional-grade document engine for creating high-fidelity, print-ready PDFs with advanced layout control and multi-language support.
Central Intelligence
Central Intelligence provides a long-term, persistent memory layer for AI agents to remember facts and recall them across sessions via semantic search.
Daily News Brief
An automated news aggregation and AI summarization tool that delivers personalized daily briefings to messaging platforms.
doc-converter
A high-fidelity document converter for seamless transitions between Word, PDF, Markdown, and HTML formats.
Featured*
