A comprehensive technical SEO skill focused on optimizing site structure, fixing crawl errors, and ensuring discoverability by both traditional search engines and AI crawlers.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install site-crawlability
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install site-crawlability using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is SEO Technical: Crawlability?
This Openclaw Skills module provides a structured approach to Technical SEO, specifically targeting site crawlability. It focuses on eliminating technical barriers that prevent search engines and AI agents from effectively indexing content. By addressing critical issues like redirect chains, broken links, and deep site hierarchies, it ensures that your website remains visible and accessible.
Furthermore, it addresses modern challenges like optimizing for AI-driven bots such as GPTBot and ClaudeBot through Server-Side Rendering (SSR) and effective URL management. Utilizing this skill ensures that your site structure is optimized for the modern web where AI search visibility is as critical as traditional search rankings.
SEO Technical: Crawlability Use Cases
- Reducing crawl budget waste on large-scale websites with high URL counts.
- Resolving orphan page issues where valuable content is not linked from the main site structure.
- Optimizing infinite scroll or masonry layouts for search engine discoverability.
- Ensuring high-performance indexing for AI-driven search engines and generative bots.
- Troubleshooting redirect loops or chains that negatively impact user experience and bot performance.
How SEO Technical: Crawlability Works
- Assess project context by reading .claude/project-context.md or .cursor/project-context.md to understand the existing site architecture.
- Conduct a technical audit of redirects, broken links (404 errors), and deep page hierarchies (pages more than 4 clicks away).
- Evaluate the development framework (e.g., Next.js, SPA) to determine rendering strategies and their impact on crawler discoverability.
- Apply optimization techniques for pagination and infinite scroll based on search-friendly recommendations.
- Verify SSR/ISR implementation to ensure critical content is available in the initial HTML for AI crawlers like GPTBot and ClaudeBot.
- Generate a prioritized list of fixes for site structure and internal linking.
SEO Technical: Crawlability Setup
To integrate this skill into your workflow, ensure your project structure is clearly defined in your local metadata files. This allows Openclaw Skills to understand your specific framework and file paths.
# Initialize project context for better skill awareness
mkdir -p .claude && touch .claude/project-context.md
Usage requires invoking the crawlability audit within your AI agent environment. Ensure your sitemap.xml and robots.txt are accessible for a full audit.
SEO Technical: Crawlability Data Schema & Taxonomy
The skill organizes its output and analysis into the following data taxonomy:
| Component | Description |
|---|---|
| Redirect Audit | A detailed list of multi-hop chains and loops requiring flattening to the final destination URL. |
| Broken Link Report | Identification of internal and external 4xx errors that need removal or 301 redirection. |
| Structural Map | Analysis of page depth relative to the homepage and identification of orphan pages. |
| Pagination Specs | Technical requirements for search-friendly scroll, including pushState and unique URL generation. |
| AI Crawler Readiness | Assessment of SSR availability and reduction of 404/redirect waste for GPTBot/ClaudeBot. |
SEO Technical: Crawlability Advanced Features
- Automated redirect chain detection and mapping for complex URL migrations.
- Strategic internal link placement recommendations to recover orphan pages.
- SSR-first validation for modern AI crawler optimization and Generative Engine Optimization (GEO).
- Cross-referencing with robots.txt and sitemap configurations to ensure crawl budget efficiency.
- Dynamic URL parameter management to prevent crawl traps in faceted navigation.
SKILL.md
Loading
Related Openclaw Skills
Auto-Report
Automate the generation and delivery of structured reports across any chat channel using scheduled cron tasks.
IndexNow
A professional tool for implementing the IndexNow protocol to ensure search engines are instantly notified of new or updated content.
SEO Technical Indexing
A specialized technical SEO skill for diagnosing and resolving Google Search Console indexing issues and managing page-level crawl directives.
SEO Technical: Mobile-Friendly
An essential developer tool for ensuring websites meet mobile-first indexing requirements and mobile usability standards.
Usage Costs
A powerful analytics tool to monitor, report, and optimize AI token usage and costs across all agent sessions.
Research Synthesizer
A high-performance research engine that executes parallel searches, deduplicates findings, and delivers cited, concise answers under 400 words.
Featured*
