A sophisticated scraping tool that utilizes real Chrome binaries and user profiles to bypass bot detection and access authenticated web content.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install browser-scraper
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install browser-scraper using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Browser Scraper?
The Browser Scraper is a specialized tool within the Openclaw Skills ecosystem designed to handle complex web scraping tasks that traditional headless browsers fail to execute. By leveraging Playwright and a real Chrome or Chromium binary, it allows AI agents to interact with websites using existing user profiles. This means the scraper can share session cookies, authentication states, and hardware fingerprints to successfully navigate sites protected by Cloudflare, Reddit, or other advanced bot detection mechanisms.
This skill is particularly effective for developers who need to extract data from authenticated sessions or capture screenshots of pages exactly as a human user would see them. It offers a bridge between automated data extraction and the high-fidelity rendering of a standard desktop browser.
Browser Scraper Use Cases
- Scraping data from platforms like Reddit or social media that aggressively block standard headless browsers.
- Extracting content from behind login walls by sharing your existing system Chrome session.
- Generating high-fidelity screenshots of complex web applications for visual auditing.
- Automating research tasks on sites that require persistent cookie storage and human-like fingerprints.
How Browser Scraper Works
- The skill initiates a Playwright instance using a persistent browser context pointed at a specific Chrome profile directory.
- It launches a real Chrome binary with automation flags disabled to prevent detection by anti-bot scripts.
- The scraper navigates to the target URL and waits for the specified CSS selectors or a default timeout to ensure content is fully rendered.
- It executes a navigator.webdriver patch and other stealth techniques to maintain a low profile during the session.
- Collected data is formatted into JSON and sent to stdout, while a screenshot is simultaneously saved for debugging or record-keeping.
Browser Scraper Setup
To get started with this skill, ensure you have Playwright and a compatible browser installed:
# Navigate to the skill directory
cd ~/.openclaw-yekeen/workspace/skills/browser-scraper/
# Install the necessary dependencies
npm install playwright
# Basic usage with system default profile
node scripts/scrape.mjs https://www.example.com
Note: Ensure your target Chrome profile is closed before running the script to avoid SingletonLock errors, or use the --profile flag to launch an isolated instance.
Browser Scraper Data Schema & Taxonomy
The Browser Scraper organizes its output and session data according to the following structure:
| Data Type | Location / Format | Description |
|---|---|---|
| Scraped Data | JSON to Stdout | Contains matched elements, attributes, or a page preview. |
| Screenshots | /tmp/browser-scraper-last.png |
A PNG capture of the most recently scraped page. |
| User Profiles | profiles/<name>/ |
Local directory for storing isolated browser cookies and cache. |
| System Profiles | OS-specific paths | Accesses default Chrome data on macOS, Linux, or Windows. |
Browser Scraper Advanced Features
- Named Profile Isolation: Create and manage separate browser profiles in the profiles/ directory for different automation tasks.
- Stealth Configuration: Automatically disables blink features and patches browser properties to mimic human behavior.
- Interactive Mode: Use the
--keep-openflag to maintain the browser session for manual troubleshooting or complex multi-step workflows. - Lazy-Load Handling: Configurable
--waitparameters to ensure asynchronous content is captured correctly. - Selector-Specific Extraction: Targeted scraping using standard CSS selectors to return precise JSON data objects.
SKILL.md
Loading
Related Openclaw Skills
CTF Malware & Network Analysis
A comprehensive toolkit for analyzing malicious binaries, deobfuscating scripts, and investigating complex network traffic in CTF environments.
CTF OSINT Skill
A comprehensive toolkit for performing open-source intelligence and investigative research during Capture The Flag competitions.
CTF AI/ML
A comprehensive toolkit for attacking and analyzing machine learning models in competitive cybersecurity environments.
csharp-dotnetcore-natasha
A powerful C# dynamic compilation skill that enables developers to generate classes, methods, and high-performance delegates at runtime using the Natasha library.
Windows API Usage Monitor
A native Windows monitoring tool designed to track, analyze, and report OpenClaw API consumption and costs through log parsing.
RR Reboot Report
A monitoring skill for Openclaw Skills that detects unexpected system reboots and sends alerts when systems come back online.
Featured*
