A specialized evaluation toolkit to quantify the esoteric knowledge density and performance benchmarks of your AI knowledge packs.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install expertpack-eval
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install expertpack-eval using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is ExpertPack Eval?
ExpertPack Eval is a sophisticated utility designed for developers using Openclaw Skills to validate the unique value of their custom knowledge bases. It addresses a core challenge in AI development: determining whether a knowledge pack actually provides new information or simply repeats what frontier models already know. By calculating the Esoteric Knowledge (EK) ratio, this skill helps you ensure that your agent frontier remains sharp and value-additive.
As a companion to the core expertpack system, ExpertPack Eval automates the testing of frontier models like GPT-4 and Claude via OpenRouter. It utilizes a blind-probing methodology to verify if a model can answer specific propositions without access to your pack, effectively measuring the 'information gap' that your Openclaw Skills are filling.
ExpertPack Eval Use Cases
- Quantifying the unique value of a custom knowledge pack compared to baseline LLM weights.
- Benchmarking agent performance across different versions of a dataset or model architecture.
- Running automated regression tests for pack-powered AI agents using LLM-as-judge scoring.
- Filtering out general knowledge padding from specialized datasets to improve RAG efficiency.
How ExpertPack Eval Works
- The skill initiates a blind probe by querying multiple frontier models via OpenRouter using propositions extracted from your pack.
- It leverages a high-tier judge model (such as Claude 3.5 Sonnet) to objectively score model responses.
- An EK ratio is calculated based on the percentage of propositions the base models could not answer correctly without the pack.
- Users can run structured evaluation sets against live agent endpoints to verify performance in real-world conversational scenarios.
- The tool generates YAML outputs that integrate with Obsidian frontmatter, allowing for long-term quality tracking using Dataview.
ExpertPack Eval Setup
To get started with this component of your Openclaw Skills, ensure you have python3 installed and an active OpenRouter API key.
# Set your OpenRouter API key (or let OpenClaw resolve it from your auth profile)
export OPENROUTER_API_KEY='your_api_key'
# Run a blind probe to measure EK ratio
python3 scripts/eval-ek.py <path-to-pack> --models gpt-4.1-mini,claude-sonnet-4-6
ExpertPack Eval Data Schema & Taxonomy
ExpertPack Eval utilizes structured YAML files to manage metadata and evaluation results, ensuring compatibility with the broader ecosystem of Openclaw Skills.
| File/Field | Purpose |
|---|---|
| manifest.yaml | Stores the measured ek_ratio, timestamp, and models tested. |
| eval-set.yaml | Contains the categorized test questions (Basic, Intermediate, Advanced). |
| results.yaml | Detailed output containing per-proposition scores and judge feedback. |
| ek_score | A decimal value (0.0 - 1.0) representing the esoteric density of the pack. |
ExpertPack Eval Advanced Features
- Multi-model benchmarking support for simultaneous testing across GPT, Claude, and Gemini.
- Automated LLM-as-judge scoring logic with configurable judge models to minimize bias.
- Obsidian-compatible frontmatter integration for professional knowledge management workflows.
- Live endpoint testing capabilities to validate agent training and structure improvements.
- Seamless workflow integration with other Openclaw Skills for end-to-end agent development.
SKILL.md
Loading
Related Openclaw Skills
Agent Memory Setup
A comprehensive setup utility for building resilient, multi-tiered memory architectures for OpenClaw agents.
Chinese De-AI Content Refiner (zh_de_ai_ify)
A specialized tool for stripping artificial AI-smell and corporate jargon from Chinese text to produce natural, human-sounding content.
Achievement-QZTC Analysis Generator
A professional tool for generating and auditing course objective achievement analysis reports from Excel data.
TickTick CLI Skill
A comprehensive command-line interface skill for managing TickTick tasks, projects, and reminders through automated workflows.
Ontology to ExpertPack Converter
A technical migration utility designed to transform append-only Ontology knowledge graphs into portable, high-fidelity ExpertPacks.
GitHub PR Writer
An AI-powered utility that generates professional, structured GitHub pull request descriptions by analyzing your branch diffs and commit history.