A specialized web scraping tool for AI agents that maintains logged-in sessions across major platforms using persistent browser contexts.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install persistent-browser
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install persistent-browser using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Persistent Browser Scraper?
Persistent Browser Scraper is a powerful utility designed for Openclaw Skills that leverages Playwright to maintain a continuous main-identity session. By utilizing a persistent user data directory, it allows AI agents to bypass the limitations of anonymous browsing, enabling them to access and extract data from platforms that typically require authentication, such as X (Twitter), GitHub, and HuggingFace.
The skill is specifically engineered to handle the complexities of modern Single Page Applications (SPAs). It operates in a non-headless mode to ensure compatibility with anti-bot measures on platforms like X, ensuring that the AI agent receives full rendered content rather than blank pages or login prompts.
Persistent Browser Scraper Use Cases
- Accessing private or personalized content on GitHub and HuggingFace.
- Monitoring real-time discussions and trends on X/Twitter and Reddit.
- Extracting metadata and transcripts from YouTube videos within a user context.
- Automating data collection from Kaggle competitions and datasets.
- Researching web-based information that requires a persistent login state.
How Persistent Browser Scraper Works
- The AI agent identifies a request that requires external web searching or specific platform access.
- The system automatically clears any existing SingletonLock in the user data directory to prevent process conflicts.
- A Playwright instance is launched using the launch_persistent_context method pointing to the main-identity profile.
- The browser navigates to the target URL in a visible (non-headless) window to satisfy platform-specific rendering requirements.
- The script implements a context-aware wait time (e.g., 8000ms for X) to allow for JavaScript hydration and SPA rendering.
- The relevant text content is extracted from the DOM and returned to the agent while the browser context is safely closed.
Persistent Browser Scraper Setup
To integrate this into your Openclaw Skills library, ensure Playwright is installed and your local profile directory is initialized.
# Install Playwright dependencies
pip install playwright
playwright install chromium
# Ensure the profile directory exists
mkdir -p /home/kncao/.openclaw/browser-profiles/main-identity
# Clear lock files before execution
rm -f /home/kncao/.openclaw/browser-profiles/main-identity/SingletonLock
Persistent Browser Scraper Data Schema & Taxonomy
The skill manages its state through a local directory structure and follows a specific timing schema for different domains:
| Platform | Wait Time (ms) | Rendering Type |
|---|---|---|
| X/Twitter | 8000 | React SPA |
| YouTube | 6000 | Lazy Loading |
| GitHub | 3000 | SSR |
| HuggingFace | 5000 | React |
| 6000 | Hybrid SSR/JS | |
| Kaggle | 8000 | React SPA |
All metadata and session cookies are stored in the main-identity folder, ensuring persistent login states across sessions.
Persistent Browser Scraper Advanced Features
- Persistent identity management via launch_persistent_context for seamless authenticated browsing.
- Automatic SingletonLock cleanup to ensure high reliability in multi-call environments.
- Stealth configuration including disabled AutomationControlled flags and no-sandbox arguments.
- Dynamic hydration delays optimized for the performance characteristics of specific high-traffic web platforms.
- Non-headless execution mode specifically tuned to bypass X/Twitter's blank-page anti-scraping logic.
SKILL.md
Loading
Related Openclaw Skills
Exa Research
An advanced web intelligence skill that enables AI agents to perform grounded searches, fetch page contents, and discover similar links with high precision.
AUSUB Gold Investment Tracking
A specialized tool for tracking Gold ETF market conditions and generating rule-based dynamic investment suggestions.
MS Forms Auto
An automated form submission engine that syncs Outlook and TMS calendars to auto-fill and submit Microsoft Forms, even with M365 MFA requirements.
Fortune-Telling Service
A comprehensive divination and fortune-telling platform providing professional metaphysical analysis using both Eastern and Western traditions.
Pesquisa Profunda
A comprehensive 8-phase research framework designed to transform raw data into verified, high-authority reports with professional citations.
Google Workspace Automation
A comprehensive automation tool for managing Gmail, Calendar, Drive, and Sheets through the Openclaw Skills framework.
Featured*
