An AI-powered utility for auditing, analyzing, and generating robots.txt files with specialized support for over 20 AI crawlers and bots.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install robotstxt-ai
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install robotstxt-ai using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is robots-ai?
robots-ai is a specialized tool designed to bridge the gap between traditional SEO and the modern AI era. It allows developers and site owners to gain granular control over how AI crawlers—such as OpenAI's GPTBot, Anthropic's ClaudeBot, and Google-Extended—interact with their web content. By integrating with Openclaw Skills, this tool simplifies the complex process of managing crawler permissions, ensuring your site's data is accessed only by the agents you approve.
Whether you need to protect proprietary data from being used in LLM training or ensure your content is discoverable by AI search engines like Perplexity, robots-ai provides the technical insights necessary to manage your site's robots.txt effectively. It helps users distinguish between standard search engine indexing and AI-specific data harvesting, which is a critical distinction in the current digital landscape of Openclaw Skills.
robots-ai Use Cases
- Auditing existing robots.txt files to identify hidden vulnerabilities or unintended AI access.
- Generating a new robots.txt file that blocks all known AI training bots while maintaining standard search engine visibility.
- Investigating whether a specific website has opted out of Anthropic or OpenAI crawling.
- Creating a centralized robots.txt configuration that includes sitemap directives and custom disallow rules for administrative paths.
How robots-ai Works
- Input a website URL or the content of an existing robots.txt file for analysis.
- The skill fetches the robots.txt and parses all User-agent directives along with Allow and Disallow rules.
- It cross-references the parsed rules against a built-in database of 20+ AI crawlers, identifying which are allowed or blocked.
- Based on user requirements (e.g., "block all AI training"), it generates a formatted robots.txt string.
- The final output includes syntax-highlighted code and explanatory comments for each section, ready for deployment.
robots-ai Setup
To get started with robots-ai within your environment, ensure you have the framework for Openclaw Skills installed.
# Standard setup for the robots-ai skill
openclaw install robots-ai
Configure your agent to recognize the bot database to provide the most accurate audits for your Openclaw Skills workflow.
robots-ai Data Schema & Taxonomy
The skill maintains a comprehensive registry of AI bots to ensure accuracy across all Openclaw Skills operations.
| Field | Description |
|---|---|
| Bot Name | The common name of the AI entity (e.g., GPTBot, ClaudeBot). |
| User-Agent | The specific string used in robots.txt directives. |
| Company | The organization responsible for the crawler (e.g., Meta, Apple). |
| Bot Type | Classification such as AI Crawler (training) or AI Search. |
Generated outputs follow the standard robots.txt protocol with grouped agent directives and sitemap metadata.
robots-ai Advanced Features
- Granular toggle for AI Search vs. AI Training: Distinguish between bots that provide traffic (PerplexityBot) and those that only crawl for training (CCBot).
- Google-Extended logic: Special handling to ensure Google Search visibility is maintained while controlling Gemini training.
- Bulk AI blocking: One-command generation to block the top 20+ known AI harvesters.
- Contextual auditing: Suggests improvements based on modern SEO best practices for Openclaw Skills users.
SKILL.md
Loading
Related Openclaw Skills
Dify Workflow DSL Expert
A specialized skill for authoring, editing, and validating Dify Workflow DSL files for self-hosted instances using an export-first approach.
Git Hooks Toolkit
A comprehensive toolkit for generating, installing, and managing production-ready Git hooks using pre-built templates for linting, testing, and branch protection.
Sprinklr
A comprehensive integration for Sprinklr that allows AI agents to manage customer interactions, marketing assets, and automated workflows across digital channels.
Liquidity Reference Agent
A zero-dependency reference tool providing instant access to essential blockchain liquidity formulas, regulations, and strategic documentation.
Pesticide Management Reference
A comprehensive reference skill for managing chemical classes, application protocols, and regulatory compliance in modern agriculture.
Webfetch
A high-performance CLI tool for fetching web content and converting it into structured Markdown, plain text, or raw HTML.
Featured*
