A specialized tool for automating the extraction of X/Twitter posts and media into organized, offline-ready Markdown files.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install crawl-from-x
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install crawl-from-x using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Crawl From X?
Crawl From X is a robust utility designed for developers and researchers who need to archive or analyze content from X (formerly Twitter). By leveraging the Browser Relay system, it bypasses traditional API limitations to fetch the latest posts from a curated list of users. This tool is a vital addition to the ecosystem of Openclaw Skills, providing a seamless bridge between social media streams and local knowledge bases.
The tool handles everything from user management to media downloading, ensuring that images, GIFs, and videos are stored locally. It automatically rewrites media URLs in the generated Markdown files to point to local relative paths, making the content fully portable and searchable within any Markdown-supported environment.
Crawl From X Use Cases
- Archiving daily updates from influential X users for personal knowledge management.
- Collecting research materials and industry insights from specific thought leaders.
- Automating the monitoring of specific accounts without manual browsing.
- Creating local, offline-ready backups of social media threads including images and videos.
How Crawl From X Works
- The user adds target X usernames to a local tracking list managed by the skill script.
- The script establishes a connection through the OpenClaw Browser Relay to a logged-in browser session.
- It navigates to the profile pages and extracts the latest posts while implementing random delays to mimic human behavior.
- Media assets such as images and videos are identified and downloaded to a dedicated local directory.
- The skill compiles the extracted text and metadata into a Markdown file, mapping all media to the downloaded local files.
Crawl From X Setup
Install the skill via the CLI:
npx clawhub@latest install crawl-from-x
Before running the crawler, ensure you have the OpenClaw Browser Relay extension installed and active in your browser. Start the relay service:
openclaw browser start
Log in to your X account in the connected browser to allow the skill to access post content.
Crawl From X Data Schema & Taxonomy
The skill organizes its data within the skill directory using the following structure:
| File/Folder | Description |
|---|---|
users.txt |
Stores the list of tracked X usernames. |
results/ |
Root directory for all exported crawl data. |
results/posts_[timestamp].md |
The generated Markdown report with local media links. |
results/images/ |
Directory containing all downloaded images, GIFs, and videos. |
results/posts_[timestamp].txt |
A list of processed URLs (generated during full crawls). |
Crawl From X Advanced Features
- Bulk User Processing: Capability to crawl an entire list of users in one command sequence.
- Media Localization: Automated replacement of remote URLs with local relative paths for offline viewing.
- Intelligent Throttling: Built-in randomized delay logic to stay within platform rate limits.
- Command-Line Management: Full CLI suite to add, list, and remove users from the tracking database.
- Openclaw Skills Integration: Designed to work natively with the Browser Relay for high-fidelity data extraction.
SKILL.md
Loading
Related Openclaw Skills
Acorn Prover Skill
A specialized tool for writing, verifying, and documenting formal mathematical and cryptographic proofs using the Acorn theorem prover.
Social Media Automator
An automated content repurposing tool that converts articles and URLs into platform-optimized social media posts and threads.
SEO Content Engine
An automated end-to-end engine for SEO research, content structuring, and high-quality draft generation.
Competitor Monitor
An automated AI intelligence tool for tracking competitor pricing, features, and marketing moves in real-time.
Jimeng AI Video Automation
A comprehensive automation solution for batch generating AI videos on the Jimeng AI platform using Playwright and session persistence.
Web Form Automation
Automate complex web form interactions and file uploads using Playwright with reliable session management and event-driven inputs.
METADATA
Featured*
