A high-performance recursive crawler that transforms entire website sections into clean Markdown via a local-first API approach.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install markdown-new-crawl
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install markdown-new-crawl using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Markdown.New Crawl?
Markdown.New Crawl is an advanced tool within the Openclaw Skills ecosystem designed for deep data extraction. It enables developers and AI agents to recursively crawl specific site sections, converting HTML content into structured Markdown. By prioritizing a local-first access model via curl, it provides a robust alternative to standard browser-based scraping.
This skill is particularly effective for documentation harvesting and knowledge base construction. It handles complex tasks like asynchronous job polling, JavaScript rendering for SPAs, and granular URL filtering, ensuring that the resulting data is clean, formatted, and ready for LLM consumption or static site generation.
Markdown.New Crawl Use Cases
- Extracting entire documentation sites for local LLM training or RAG pipelines.
- Automating recursive site audits to gather content in a unified Markdown format.
- Performing job-based crawl polling to monitor long-running extraction tasks.
- Converting complex single-page applications (SPAs) into readable text using JS rendering.
How Markdown.New Crawl Works
- Initiate a crawl by sending a POST request to the API with a target URL and optional parameters like depth or page limits.
- Receive a unique Job ID from the service to track the progress of the asynchronous operation.
- Poll the status endpoint using the Job ID to monitor the crawl until completion.
- Retrieve the final payload, which defaults to concatenated Markdown or structured JSON if requested.
- Optionally cancel ongoing jobs or manage cached results through dedicated management endpoints.
Markdown.New Crawl Setup
To integrate this into your Openclaw Skills workflow, use standard terminal utilities. No complex authentication is required for basic operations.
# Start a new crawl job
curl -X POST "https://markdown.new/crawl" \
-H "Content-Type: application/json" \
-d '{"url":"https://docs.example.com","limit":50}'
# Check job status
curl "https://markdown.new/crawl/status/<jobId>"
Markdown.New Crawl Data Schema & Taxonomy
The skill organizes data based on crawl jobs, offering flexible output formats for different integration needs.
| Feature | Specification |
|---|---|
| Default Format | Concatenated Markdown documents |
| Structured Format | JSON per-page records (via ?format=json) |
| Retention | 14 days for job IDs and associated data |
| Page Limit | 1-500 pages per individual crawl job |
| Depth | 1-10 levels of recursive link following |
Markdown.New Crawl Advanced Features
- JavaScript Rendering: Support for dynamic content extraction in modern web apps.
- Granular Filtering: Use includePatterns and excludePatterns wildcards to target specific site sections.
- Cache Management: Control freshness with maxAge parameters (0 to 604,800 seconds).
- Source Strategies: Choose between link following, sitemap discovery, or a hybrid approach.
SKILL.md
Loading
Related Openclaw Skills
Halo CLI Shared
A foundational skill for navigating the Halo CLI ecosystem, providing shared rules for profiles, JSON outputs, and command routing.
BET (Safety and Health Plan) Generator
A specialized tool for generating legally compliant Safety and Health Plans (BET) for Hungarian construction projects.
Crypto Funding Rate Harvester
An automated scanner that identifies high-yield delta-neutral carry trade opportunities across major crypto exchanges.
AI Video Generation Skill
An end-to-end automated pipeline for generating professional-grade videos from text using leading AI models and FFmpeg editing.
Five: Cultural Insights and Project Matrix
A dual-purpose skill providing deep cultural analysis of the number 5 alongside the technical roadmap of the OpenClaw ecosystem.
Community Group Buying Product Selection
A comprehensive strategy skill for optimizing product selection and pricing lifecycles in community group buying ecosystems.
Featured*
