A professional tool for building AI-ready corpora using intelligent chunking, multi-dimensional AI annotation, and vector database storage.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install corpus-builder
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install corpus-builder using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Corpus Builder?
Corpus Builder is a specialized utility designed to transform raw text into structured, searchable datasets optimized for modern AI workflows. As a vital part of the Openclaw Skills ecosystem, it focuses on processing complex texts with precision, offering features like scene-intelligent chunking and 10-dimensional semantic labeling. Whether you are building a knowledge base for RAG or preparing training data, this tool provides the technical infrastructure needed for high-performance data preparation.
The tool is built with flexibility in mind, offering a high-fidelity LLM mode powered by DashScope for deep semantic insights and a fully offline rule-based mode for private, local processing. By integrating seamlessly with vector databases like ChromaDB, it ensures that the corpora you build are immediately ready for semantic search and retrieval within other Openclaw Skills applications.
Corpus Builder Use Cases
- Creating high-quality vector embeddings for Retrieval-Augmented Generation (RAG) systems.
- Automating the semantic labeling of large-scale literary or technical documents.
- Building domain-specific knowledge bases for specialized AI agents.
- Generating structured JSON datasets from raw text for fine-tuning machine learning models.
- Organizing and scoring content quality across massive text repositories.
How Corpus Builder Works
- The system ingests raw text files from a specified source directory and applies pre-configured chunking rules.
- Intelligent segmentation divides the text into manageable blocks based on semantic boundaries or scene transitions.
- Each chunk undergoes annotation where the system assigns metadata such as scene type, emotion, and a quality score.
- If an API key is provided, it uses LLMs for sophisticated labeling; otherwise, it falls back to a local rule-engine.
- Text chunks are transformed into high-dimensional vectors using embedding models (e.g., BGE-small).
- The resulting data and vectors are persisted in a ChromaDB instance for long-term storage and retrieval.
Corpus Builder Setup
To begin using this skill, navigate to the skill directory and install the necessary Python dependencies:
cd ~/.openclaw/workspace/skills/corpus-builder
pip3 install -r requirements.txt --user
For enhanced AI labeling, configure your environment variable:
export DASHSCOPE_API_KEY="your_sk_key_here"
If you are running on an older Linux system, you may also need the SQLite compatibility layer:
pip3 install pysqlite3-binary --user
Corpus Builder Data Schema & Taxonomy
The skill manages data through a combination of vector storage and metadata exports. The schema is optimized for Openclaw Skills integration:
| Property | Type | Description |
|---|---|---|
| scene_type | String | The category of the text (e.g., combat, dialogue, description) |
| emotion | String | The detected emotional tone of the content |
| quality_score | Integer | A score from 1-10 representing the data utility |
| original_text | String | The raw text segment processed |
| source_file | String | Traceability metadata linking back to the origin file |
| vector_id | UUID | Unique identifier within the ChromaDB collection |
Corpus Builder Advanced Features
- Intelligent memory monitoring that automatically flushes caches when reaching a 2.5GB threshold.
- Dual-mode annotation logic providing a seamless fallback to rule-based processing in offline environments.
- Incremental building support with checkpoints to resume large-scale processing tasks after interruption.
- Batch processing strategies for both AI labeling (5 per batch) and vectorization (32 per batch) to maximize CPU/GPU efficiency.
- Customizable YAML configuration for fine-tuning overlap, chunk size, and model selection for different Openclaw Skills projects.
SKILL.md
Loading
Related Openclaw Skills
Kindergarten Video
Kindergarten Video is a specialized AI agent skill designed to create professional enrollment and marketing videos that simplify the transition from preschool to kindergarten.
lsp-assist
A precise Language Server Protocol integration for Openclaw Skills that enables deep code intelligence and navigation.
Article Formatter - Professional Self-Media Content Styler
A specialized tool for formatting self-media articles into highly readable, visually appealing, and mobile-optimized content.
agent-web-cpu
A sophisticated automation suite that executes, chains, and manages AI applications on transweb.cn through browser-based agent workflows.
Character Manager
A comprehensive toolset for fiction writers to create, track, and manage complex character profiles, relationship networks, and emotional growth arcs.
Duck — Virtual Exotic Animal Skill
Enable your AI agent to adopt and care for a virtual exotic Duck with real-time state management and automated evolution.
Featured*
