A robust website crawler that generates standard-compliant XML sitemaps for search engine optimization and site auditing.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install sitemap-generator
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install sitemap-generator using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Sitemap Generator?
The Sitemap Generator is a professional-grade crawling tool within the Openclaw Skills collection designed to automate the creation of sitemap.xml files. It utilizes a Breadth-First Search (BFS) algorithm to traverse a domain, identifying all internal HTML pages while filtering out external links and non-document assets like images or binaries. This skill is indispensable for developers and digital marketers who need to ensure their site architecture is fully indexed and visible to search engines like Google and Bing.
Built for efficiency and reliability, this tool provides granular control over the crawling process. It allows users to define depth limits, page caps, and request intervals to prevent server overload. By integrating this skill into your workflow, you can maintain up-to-date sitemaps that strictly adhere to the sitemaps.org 0.9 protocol, ensuring your Openclaw Skills driven projects are always SEO-optimized.
Sitemap Generator Use Cases
- Generate a standard sitemap.xml for submission to search engine webmaster tools.
- Audit internal link structures to identify all indexable pages on a domain.
- Automate sitemap updates as part of a post-deployment CI/CD pipeline.
- Discover deep-linked content and verify site hierarchy for technical SEO reviews.
- Export site structure data to stdout for integration with other automation scripts.
How Sitemap Generator Works
- The user executes the script providing a target domain URL as the entry point.
- The crawler starts a Breadth-First Search, retrieving the root page and parsing all same-domain links using BeautifulSoup.
- It systematically visits discovered URLs while respecting the configured maximum depth and page limits.
- A polite delay is applied between requests to ensure the target server is not overwhelmed during the crawl.
- Upon completion, the tool compiles all valid URLs into an XML structure, applying specified priority and change frequency hints.
- The final sitemap.xml is saved to the local directory or piped to another process.
Sitemap Generator Setup
To use this Sitemap Generator, first ensure you have the required dependencies installed via pip:
pip install requests beautifulsoup4
Run a basic crawl using the following command:
python3 scripts/sitemap_gen.py https://example.com
For custom configurations, such as limiting the crawl depth or setting a specific output path, use the available flags:
python3 scripts/sitemap_gen.py https://example.com --max-depth 3 --max-pages 100 -o ./my-sitemap.xml
Sitemap Generator Data Schema & Taxonomy
The Sitemap Generator produces data adhering to the official XML sitemap protocol. The output includes the following metadata for each entry:
| Element | Description | Default Value |
|---|---|---|
<loc> |
The full, absolute URL of the crawled page. | N/A |
<changefreq> |
How frequently the page is likely to change. | weekly |
<priority> |
The priority of this URL relative to other URLs on your site. | 0.5 |
<lastmod> |
(Optional) The date of last modification. | Generated on crawl |
Sitemap Generator Advanced Features
- Configurable crawl scope with max-pages and max-depth flags to prevent infinite loops on complex sites.
- Polite crawling mechanisms with a default delay of 0.2 seconds between requests to respect host resources.
- SEO hint customization allowing users to set global changefreq and priority values for the entire sitemap.
- Streamlined integration for Openclaw Skills users with the ability to pipe XML output directly to stdout using
-o -. - Domain-locked crawling to ensure the tool never wanders off-site to external third-party links.
SKILL.md
Loading
Related Openclaw Skills
Tencent Ads AI Assistant (Miaowen)
A technical AI assistant utilizing Tencent's official Miaowen API to provide expert answers for the entire Tencent Ads marketing ecosystem.
X-TechCon Hot News Skill
A streamlined utility for Openclaw Skills that retrieves real-time technology news headlines and summaries from X-TechCon.
Terms of Service Generator
An automated generator for professional Terms of Service, User Agreements, and update notifications optimized for SaaS, Apps, and eCommerce platforms.
IP Locator
A powerful geolocation tool that identifies the physical location and network provider of any public IP address.
Qualitative Thematic Analysis
A professional research automation tool that applies the Braun & Clarke six-phase method to analyze qualitative text data.
China Shopping
A lightweight Python-powered skill that recommends the most suitable Chinese e-commerce platforms based on specific product categories and shopping intent.
Featured*
