A technical benchmarking tool that measures whether Skill modifications actually improve AI performance on real-world tasks using automated pass/fail criteria.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install auto-improvement-evaluator
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install auto-improvement-evaluator using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Improvement Evaluator?
The Improvement Evaluator is a core component within the Openclaw Skills ecosystem designed to move beyond subjective quality assessments. While other tools focus on document structure or scoring, this skill executes a predefined suite of tasks to determine an objective execution pass rate. It allows developers to verify that a SKILL.md change results in tangible performance gains by comparing candidate versions against a established baseline.
By running AI tasks through different judging mechanisms—such as keyword matching, Pytest suites, or LLM-based rubrics—the evaluator provides a concrete delta in performance. This ensures that every iteration of an AI agent is backed by empirical data, preventing regressions and guiding the generator toward targeted fixes based on actual execution failures.
Improvement Evaluator Use Cases
- Verifying that SKILL.md updates improve real-world AI task execution rather than just documentation syntax.
- Running a standardized task suite against candidate skills to compare performance against a baseline.
- Generating execution pass rate metrics to serve as a quality gate for CI/CD pipelines.
- Discovering baseline failures in current skill implementations to identify areas for improvement.
How Improvement Evaluator Works
- The evaluator loads a YAML-based task suite containing prompts and specific judge criteria.
- In Pipeline mode, it receives a candidate skill and checks if its previous discriminator score meets the evaluation threshold.
- It executes the tasks using the AI agent, applying the specified judge (Contains, Pytest, or LLM-Rubric) to each output.
- It calculates the execution pass rate for both the candidate and the baseline (utilizing caching where possible).
- It outputs a JSON artifact containing the pass rate, the delta improvement, and a final verdict for the next stage in the workflow.
Improvement Evaluator Setup
To use this skill within your Openclaw Skills environment, ensure you have the required scripts and task suites ready. Use the following commands for execution:
# Pipeline mode: Compare a candidate against the baseline
python3 scripts/evaluate.py --input ranking.json --candidate-id cand-01-docs \
--task-suite tasks.yaml --state-root ./state
# Standalone mode: Directly evaluate your current SKILL.md
python3 scripts/evaluate.py --standalone --task-suite tasks.yaml \
--state-root ./state --skill-path /path/to/skill
Improvement Evaluator Data Schema & Taxonomy
The skill relies on a structured YAML task suite and produces a detailed JSON artifact.
Task Suite Schema (YAML)
| Field | Description |
|---|---|
skill_id |
Unique identifier for the skill being tested |
tasks |
A list of task objects including id, prompt, and judge |
judge |
Defines the validation logic: contains, pytest, or llm-rubric |
Output Artifact (JSON)
| Key | Description |
|---|---|
execution_pass_rate |
The percentage of tasks the candidate passed |
baseline_pass_rate |
The percentage of tasks the baseline version passed |
delta |
The net improvement or regression in performance |
verdict |
The final status (pass, fail, or skipped) |
Improvement Evaluator Advanced Features
- Multi-Judge Support: Toggle between deterministic keyword checks, structured Pytest validations, and semantic LLM rubrics.
- Intelligent Caching: Uses SHA256 hashing of skill content and suite paths to avoid redundant baseline runs.
- Conditional Evaluation: Automatically skips candidates that fall below a specific scoring threshold or aborts if the baseline is broken.
- Regression Gating: Integrates with improvement-gate to prevent the deployment of skills that decrease performance delta.
SKILL.md
Loading
Related Openclaw Skills
Zhang Lei Thinking Operating System
An advanced AI cognitive framework modeled after Zhang Lei's value investment philosophy to guide long-term business and investment decision-making.
Hillhouse Investment Framework Skill
An advanced AI-driven analytical framework based on Hillhouse Capital's philosophy to identify long-term value and evaluate business fundamentals.
QEntity
QEntity provides AI agents with complete financial personhood, allowing them to autonomously earn, hold, and spend money across global payment rails.
VCF Log Insight MCP Explorer
A Model Context Protocol server that empowers AI agents to search and analyze VMware Aria Operations for Logs natively.
HTTP Requests
A Python-powered skill for executing complex HTTP requests without the escaping and quoting issues common in traditional curl commands.
Yufeng A-Stock Quantitative Screener
A sophisticated AI-powered quantitative stock screening assistant designed for A-share short-term trading opportunities.
Featured*
