A technical SEO skill for configuring, auditing, and optimizing robots.txt files to manage search engine visibility and AI crawler access.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install robots-txt
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install robots-txt using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is robots-txt?
The robots-txt skill for Openclaw Skills is a specialized tool designed to help developers and SEO professionals manage how search engines and AI agents interact with their websites. It focuses on crawl control, providing a framework to configure Disallow and Allow directives, Sitemap declarations, and AI-specific crawler strategies. By leveraging Openclaw Skills, you can ensure that critical assets like CSS and JS remain accessible to modern search engines while protecting sensitive directories or preventing content from being used in unauthorized AI model training.
This skill bridges the gap between simple text file editing and strategic technical SEO. It provides deep insights into the Robots Exclusion Protocol (RFC 9309) and helps users distinguish between crawling restrictions in robots.txt and indexing restrictions via noindex tags. Using Openclaw Skills allows for a standardized, error-free approach to managing site accessibility in an era where AI bot traffic is rapidly increasing.
robots-txt Use Cases
- Creating a production-ready robots.txt file that follows modern SEO best practices.
- Blocking AI training crawlers like GPTBot or ClaudeBot while allowing AI search agents through Openclaw Skills.
- Auditing existing site configurations to fix accidental blocks of critical rendering assets like Next.js static files.
- Configuring specific directives for Yandex using Clean-param to optimize URL indexing.
- Protecting staging environments or API routes from being crawled by public search engines.
How robots-txt Works
- The skill first checks for project context in files like project-context.md to identify the site URL and specific indexing goals.
- It evaluates the current crawling strategy, distinguishing between path-level control and page-level indexing needs.
- A customized AI crawler strategy is developed based on whether you want to allow search-specific bots or block general training crawlers via Openclaw Skills.
- The skill identifies critical paths that must remain unblocked, such as CSS, images, and JavaScript, to ensure proper search engine rendering.
- It generates a valid, UTF-8 encoded robots.txt file structure including Sitemap references and specific User-agent rules.
robots-txt Setup
To implement this skill within your project environment using Openclaw Skills, follow these basic steps for configuration.
# Navigate to your project root or public directory
cd public/
# Create a robots.txt file if it does not exist
touch robots.txt
# Review your project-context.md to ensure the Site URL is defined
cat .claude/project-context.md
Ensure that the resulting robots.txt is served at the root of your domain (e.g., https://example.com/robots.txt).
robots-txt Data Schema & Taxonomy
The robots-txt skill organizes technical SEO directives into a structured format based on the following metadata and schema standards provided by Openclaw Skills:
| Directive | Data Type | Description |
|---|---|---|
| User-agent | String | Identifies the specific crawler the rule applies to (e.g., Googlebot, GPTBot) |
| Disallow | Path String | Defines directory or file prefixes that should not be crawled |
| Allow | Path String | Overrides Disallow rules for specific sub-paths |
| Sitemap | Absolute URL | Points search engines to the XML sitemap index |
| Clean-param | Parameter List | Used for Yandex to identify parameters that don't change page content |
All configurations are generated as UTF-8 plain text to comply with RFC 9309.
robots-txt Advanced Features
- Multi-agent AI strategy: Differentiates between search-centric AI (OAI-SearchBot) and training-centric AI (CCBot) using Openclaw Skills.
- Rendering protection: Prevents common mistakes like blocking /_next/ paths which can break SEO for modern JavaScript frameworks.
- Indexing vs. Crawling Logic: Integrates logic to ensure that pages with noindex tags are actually reachable by crawlers to process the directive.
- Project Context Integration: Automatically pulls site-specific goals from your existing project documentation for tailored robots.txt output.
- Legacy Support: Includes specific directives for older or niche search engines like Yandex and AppleBot.
SKILL.md
Loading
Related Openclaw Skills
FakeX-Till-You-AI
An intelligent posting layer that converts daily AI digests into personalized X post drafts with automated or semi-automated scheduling options.
AK Data Daily Timeout Report
A comprehensive diagnostic skill for AK Data that unifies task and crawler timeout reporting with automated log enrichment and gap analysis.
Branding Strategy & Narrative
A comprehensive framework for defining brand purpose, storytelling arcs, voice guidelines, and high-level visual strategy to ensure consistent identity across all touchpoints.
Brand Protection
A comprehensive framework for identifying, reporting, and preventing brand impersonation, phishing sites, and trademark infringement.
SEO Technical: Sitemap
A technical SEO skill designed to automate the creation, auditing, and optimization of XML sitemaps for maximum search engine visibility.
Brand Monitoring
A proactive brand monitoring skill to detect trademark infringement, impersonation, and brand abuse across digital channels.
Featured*
