A high-performance web scraping skill powered by the Zig-based Lightpanda engine for rapid headless browser data extraction.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install lightpanda-scraper
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install lightpanda-scraper using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Lightpanda Scraper?
Lightpanda Scraper is a specialized tool within the Openclaw Skills library designed for developers and security researchers who need the rendering capabilities of a headless browser without the overhead of Chromium. Built on the Zig-compiled Lightpanda engine, it achieves page load times of approximately 0.5 seconds, making it up to 90 times faster than traditional Playwright or Puppeteer setups. This skill is ideal for massive-scale link extraction and content harvesting where performance and low resource consumption are critical.
Lightpanda Scraper Use Cases
- Fast OSINT reconnaissance and page dumping for rapid threat intelligence analysis.
- Bulk link extraction and endpoint discovery during large-scale bug bounty hunting.
- Automated content scraping to generate clean markdown for LLM training data and knowledge bases.
- Anonymous crawling and data collection using integrated SOCKS5 or Tor proxy support.
How Lightpanda Scraper Works
- The user provides a target URL along with specific extraction flags via the CLI script.
- The skill invokes the Lightpanda binary, which is a lightweight Zig-based headless browser engine.
- Lightpanda fetches the page, renders the DOM, and executes any provided JavaScript snippets to ensure dynamic content is captured.
- The rendering engine transforms the processed page into the requested format, such as Markdown, raw HTML, or a list of links.
- The resulting data is streamed to the output or saved directly to a designated file for further processing within Openclaw Skills workflows.
Lightpanda Scraper Setup
To integrate this capability into your environment, install the Lightpanda binary and ensure python3 is available:
mkdir -p ~/.local/bin
curl -L https://github.com/nicholasgasior/lightpanda-browser/releases/latest/download/lightpanda-linux-x86_64 -o ~/.local/bin/lightpanda
chmod +x ~/.local/bin/lightpanda
Lightpanda Scraper Data Schema & Taxonomy
The skill organizes extracted data into clear, actionable formats based on the selected mode:
| Data Type | Flag | Description |
|---|---|---|
| Markdown | Default | Clean, readable text content perfect for AI analysis. |
| HTML | --html | The raw, rendered source code of the target page. |
| Links | --links | A categorized list of all URLs found within the document. |
| JS Result | --js | The output of a specific JavaScript evaluation on the page. |
Lightpanda Scraper Advanced Features
- MCP Server Support: Native compatibility with the Model Context Protocol for seamless AI agent integration.
- CDP Server Mode: Capability to start a Chrome DevTools Protocol server for programmatic remote access.
- Resource Stripping: Advanced options to strip CSS, images, or JS to maximize scraping velocity and reduce bandwidth.
- Network Flexibility: Full support for HTTP and SOCKS5 proxies, facilitating secure routing through Tor or VPNs.
SKILL.md
Loading
Related Openclaw Skills
OpenClaw Browser
A managed browser system for executing real-world website interactions through isolated profiles and snapshot-based automation.
Bug Bounty Report Template Generator
A specialized tool for creating professional, platform-ready bug bounty reports with automated CWE mapping and severity scoring.
recon-quick
A high-performance reconnaissance utility for automated OSINT, subdomain enumeration, and vulnerability probing using bbot and nmap.
Reboot Checker
A security-focused monitoring tool that detects unexpected system restarts and maintains a persistent boot history log.
9kvr-panorama
A comprehensive VR panorama management skill for handling everything from asset uploads and hotspot configuration to audio integration and project scoring.
Project Aura: The Emo-Agent-Toolkit
A lightweight emotional presentation layer for AI companions using RLHF and dynamic weighted algorithms.
METADATA
Featured*
