A high-performance scraping engine that uses Docker and Crawlee to extract clean, structured data from complex websites like YouTube and X/Twitter.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install deep-scraper
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install deep-scraper using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Deep Scraper?
The Deep Scraper skill is a specialized engineering tool designed for deep web extraction within the Openclaw Skills ecosystem. It utilizes a containerized Docker environment powered by Crawlee and Playwright to navigate and penetrate protections on complex, JavaScript-heavy websites. By providing interception-level raw data, this skill allows developers to bypass standard scraping hurdles and retrieve high-fidelity information from platforms like YouTube and X/Twitter.
Built with an alpha-focused philosophy, it automatically strips away advertisements and UI noise, delivering pure data optimized for processing by large language models. This makes it an essential component for any developer building advanced AI agents that require real-time, accurate web data through Openclaw Skills.
Deep Scraper Use Cases
- Extracting full video transcripts and metadata from YouTube for content analysis.
- Scraping public posts and engagement data from social media platforms like X/Twitter.
- Gathering clean text content from dynamic, single-page applications for LLM context.
- Integrating automated data collection workflows into broader Openclaw Skills deployments.
How Deep Scraper Works
- The user triggers the skill by providing a target URL through the command-line interface.
- The system invokes a Docker container based on the clawd-crawlee image to ensure an isolated and consistent environment.
- The skill uses Playwright via Crawlee to load the target page, handling all JavaScript execution and dynamic content rendering.
- The main_handler.js script processes the page, validating specific identifiers such as YouTube Video IDs to maintain data integrity.
- Extracted data is cleaned of non-essential noise and returned as a structured JSON string to stdout, facilitating easy integration with other Openclaw Skills.
Deep Scraper Setup
To get started with this skill, ensure Docker is installed and running on your host machine. Follow these steps to build and run the environment:
- Build the Docker image:
docker build -t clawd-crawlee skills/deep-scraper/
- Run the scraper using the standard CLI interface:
docker run -t --rm -v $(pwd)/skills/deep-scraper/assets:/usr/src/app/assets clawd-crawlee node assets/main_handler.js [TARGET_URL]
Ensure the assets directory is correctly mapped to allow the container to access the handler scripts within the Openclaw Skills folder structure.
Deep Scraper Data Schema & Taxonomy
The Deep Scraper skill returns results in a standardized JSON format. This allows Openclaw Skills to parse the data consistently regardless of the target source.
| Property | Type | Description |
|---|---|---|
| status | String | Execution result: SUCCESS, PARTIAL, or ERROR |
| type | String | Content category: TRANSCRIPT, DESCRIPTION, or GENERIC |
| videoId | String | The validated unique identifier (specifically for YouTube) |
| data | String | The core extracted text content or transcript data |
Deep Scraper Advanced Features
- Interception-level data capture designed to bypass common anti-bot protections on major social platforms.
- Strict ID validation logic to prevent cache contamination and ensure data accuracy within Openclaw Skills.
- Automated noise reduction that filters out advertisements, navigation links, and unrelated UI elements.
- Containerized architecture ensuring the scraper remains self-contained and portable across different environments.
SKILL.md
Loading
Related Openclaw Skills
Pocket Money
Provide your AI agent with dedicated crypto wallets on the Base network to manage USDC budgets for autonomous tasks.
Markdown Publish & Share
An API-driven utility to instantly publish Markdown content into rendered, shareable web documents with support for diagrams and math.
AutEng Docs
A professional publishing tool for technical markdown documents featuring Mermaid diagrams, KaTeX math, and persistent versioned workspaces.
AutEng Docs Curl Publish
A streamlined utility for programmatically publishing Markdown documents and diagrams to shareable web URLs via simple curl commands.
Awesome Skills Explorer
A comprehensive discovery engine for finding and installing the best community-driven Openclaw Skills for your AI agent.
OptionsHawk
OptionsHawk provides advanced options market intelligence, including flow analysis, unusual activity detection, and strategy evaluation for equities and ETFs.
Featured*
