A specialized tool for measuring real-world AI performance improvements using execution-based task suites instead of simple document structure checks.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install improvement-evaluator
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install improvement-evaluator using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Improvement Evaluator?
The Improvement Evaluator is a critical component for developers working with Openclaw Skills who need to verify that changes to a skill actually lead to better task performance. Unlike structural scoring which only looks at formatting, this tool runs actual AI tasks defined in YAML suites to calculate a concrete execution pass rate. It ensures that your AI agents are getting smarter, not just better documented, by comparing candidate skills against baseline performance metrics.
By focusing on execution rather than just documentation structure, it provides a high-fidelity signal for whether an update should be accepted. It is designed to work within an automated pipeline, helping teams maintain high quality standards for their Openclaw Skills through data-driven performance analysis.
Improvement Evaluator Use Cases
- Verifying that SKILL.md updates improve real-world AI execution rather than just document structure
- Performing regression testing between different versions of a skill to ensure performance stability
- Validating new task suites to ensure they provide a sane baseline for future evaluations
- Automating gating decisions in CI/CD pipelines based on execution pass rate metrics
- Debugging low-performing AI agents by inspecting detailed per-task failure logs and durations
How Improvement Evaluator Works
- The evaluator loads a YAML-based task suite containing 5-10 specific tasks, prompts, and judging criteria.
- It initializes the environment, identifying the candidate skill version and checking for existing baseline data in the cache.
- The skill executes the defined tasks against the AI model, monitoring for timeouts and output constraints.
- Each output is evaluated by a designated judge—choosing between ContainsJudge, PytestJudge, or LLMRubricJudge—to determine a pass or fail.
- It calculates the execution pass rate for both the candidate and the baseline, determining the performance delta.
- A final JSON artifact is generated containing the results, which is then used by the improvement-gate to make an final accept or reject decision.
Improvement Evaluator Setup
To integrate this into your workflow for Openclaw Skills, you can run the evaluator in standalone mode or as part of a pipeline. Use the following bash command for standalone evaluation during development:
python3 scripts/evaluate.py \
--standalone \
--task-suite task_suites/target-skill/task_suite.yaml \
--skill-path ./skills/target-skill \
--state-root /tmp/eval-state
Ensure your task_suite.yaml follows the version 1.0 schema, including a unique skill_id and a list of tasks with valid judge configurations.
Improvement Evaluator Data Schema & Taxonomy
The Improvement Evaluator utilizes structured YAML for inputs and generates detailed JSON artifacts for downstream consumption.
| Component | Format | Key Attributes |
|---|---|---|
| Task Suite | YAML | skill_id, version, tasks (id, prompt, judge, timeout_seconds) |
| Output Artifact | JSON | execution_pass_rate, baseline_pass_rate, delta, verdict, candidate_results |
| Judge Configuration | YAML | type (contains, pytest, llm-rubric), expected values, or rubric text |
| Evaluation State | Directory | Stores per-run logs, cached baselines, and temporary execution outputs |
Improvement Evaluator Advanced Features
- Multi-judge support allows for deterministic keyword checks, complex Python unit tests, or semantic LLM scoring
- Intelligent baseline caching with a 7-day TTL reduces API costs by avoiding redundant baseline executions
- Mock execution mode enables local testing of the evaluation pipeline without incurring LLM provider charges
- Pipeline mode integration allows it to consume ranking artifacts directly from the improvement-discriminator stage
- Path-traversal guards and timeout controls ensure secure and predictable execution within the Openclaw Skills ecosystem
SKILL.md
Loading
Related Openclaw Skills
Human-Like Memory
A persistent memory system that enables AI agents to search, recall, and store long-term context, user preferences, and historical decisions.
Humanizer: AI Text De-patterning & Humanization
A specialized skill for Openclaw Skills that identifies and removes robotic writing markers to make AI-generated content sound authentically human.
Steve Jobs Perspective
A high-fidelity mental model simulation of Steve Jobs designed to audit products, strategy, and design through the lens of technical and humanistic excellence.
Dandan Test Skill
A diagnostic utility for verifying the publishing lifecycle and CLI functionality of AI agent skills.
PLC Industrial Automation Skill
A comprehensive development skill for IEC 61131-3 PLC programming, debugging, and multi-vendor industrial automation logic.
PubMed Literature Search & Analysis
An advanced clinical medicine literature retrieval system that integrates PubMed data with JCR impact factors for high-precision research analysis.
Featured*
