A Python-based utility for creating SEO-compliant sitemap.xml and robots.txt files via web crawling or local directory scanning.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install xml-sitemap-generator
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install xml-sitemap-generator using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Sitemap Generator?
The Sitemap Generator is a specialized tool designed to simplify search engine optimization tasks within the Openclaw Skills framework. It allows developers to programmatically generate sitemaps by either crawling a live website using a Breadth-First Search (BFS) algorithm or by scanning a local project directory. By automating the discovery of pages and the generation of properly formatted XML, this skill ensures that web content is easily discoverable by search engines without the need for manual maintenance.
Built to be lightweight and dependency-free, this utility supports various output formats including XML, JSON, and plain text. It is particularly useful for modern web development workflows where sitemaps need to be updated frequently during the build process. When used as part of a suite of Openclaw Skills, it provides a robust foundation for automated SEO auditing and site indexing management.
Sitemap Generator Use Cases
- Generating a sitemap.xml for a production website to improve search engine indexing.
- Scanning a local build folder (e.g., a static site generator output) to create sitemaps before deployment.
- Creating a compliant robots.txt file that automatically references the generated sitemap.
- Exporting a site's structure to JSON format for data analysis or migration planning.
- Automating SEO metadata updates within a CI/CD pipeline using Openclaw Skills.
How Sitemap Generator Works
- The user provides a target entry point, which can be a live URL for crawling or a local directory path for file scanning.
- In crawl mode, the tool performs link extraction and normalization, ensuring only same-domain URLs are collected.
- In local mode, the tool scans for HTML, Markdown, and PHP files, using file system metadata to determine modification dates.
- The system filters out non-essential assets like images, CSS, and JavaScript to keep the sitemap clean.
- The tool assembles the discovered URLs into a valid XML structure following the sitemaps.org schema.
- Optionally, a robots.txt file is generated with standard Allow/Disallow directives and a sitemap link.
Sitemap Generator Setup
The Sitemap Generator requires Python 3.6 or higher and uses only the standard library, making it highly portable. To use this within your Openclaw Skills environment, execute the script via the CLI.
# Generate a sitemap by crawling a website
python3 scripts/sitemap_gen.py https://example.com --output sitemap.xml
# Generate a sitemap from a local directory
python3 scripts/sitemap_gen.py --local ./dist --base-url https://example.com --output sitemap.xml
Sitemap Generator Data Schema & Taxonomy
The tool generates standard SEO files and can also output metadata in structured formats. Below is the mapping of how data is handled:
| Attribute | Description | Logic |
|---|---|---|
| loc | The absolute URL of the page | Crawled link or Base-URL + local path |
| lastmod | The last modification date | Extracted from file system mtime in local mode |
| changefreq | Frequency of updates | Not included by default to follow modern SEO best practices |
| format | Output file type | Supports .xml (default), .json, or .txt |
Sitemap Generator Advanced Features
- Smart URL normalization that removes fragments and standardizes trailing slashes to prevent duplicate content issues.
- Flexible output options allowing users to receive a human-readable summary alongside the raw XML.
- Customizable crawl limits, including max-pages and timeout settings, to prevent over-taxing servers.
- Automatic detection of lastmod dates from local file systems, providing more accurate metadata than simple crawlers.
- Seamless integration for Openclaw Skills users looking to automate the creation of robots.txt for complex site architectures.
SKILL.md
Loading
Related Openclaw Skills
Incident Postmortem
A professional tool for generating structured, blame-free incident postmortem reports through automated log parsing and timeline reconstruction.
GitHub Intelligent Code Review & CI/CD Automation
A comprehensive AI-powered workflow that automates GitHub pull request reviews and intelligently generates optimized CI/CD configurations.
Agent Insight Collection
An automated intelligence gathering tool that extracts actionable business insights and churn risks from customer-facing AI agents.
Creator Collaboration Planner
A strategic planning tool for designing creator partnerships and outreach workflows across major social platforms without requiring live database access.
AI Video Generator Free Best
A high-performance AI video generation skill that converts text prompts and product images into professional 1080p marketing videos in under two minutes.
Content Repurposer
Automatically transform articles, blog posts, and videos into high-engagement social media content across multiple platforms.
Featured*
