A specialized tool for analyzing AI skill source files and generating fine-grained, scope-aware judge definitions for automated evaluation frameworks.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install generate-judgements
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install generate-judgements using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Generate Judgements for Skill Evaluation?
The Generate Judgements skill is an essential utility for developers building with Openclaw Skills who need to ensure their agents behave reliably. It automates the complex process of creating test configurations for the mlflow-skills evaluation framework by analyzing a skill's SKILL.md and supporting reference files.
By parsing workflow steps, behavior rules, and output requirements, this skill identifies distinct execution paths (scopes) and generates a comprehensive suite of binary (yes/no) questions. These questions serve as the logic for LLM judges to evaluate execution traces, ensuring every technical requirement and guideline is met during automated testing.
Generate Judgements for Skill Evaluation Use Cases
- Creating a new YAML test configuration for a newly developed agent skill.
- Updating existing judge definitions when a skill's workflow or output format changes.
- Generating regression tests that specifically target conditional branches and edge cases.
- Standardizing evaluation criteria across a suite of different Openclaw Skills.
How Generate Judgements for Skill Evaluation Works
- Analysis: The skill reads the target directory to analyze the SKILL.md, reference files, and READMEs to extract mandatory behaviors and output schemas.
- Scope Inference: It identifies distinct logical paths (e.g., a 'checklist' path vs. an 'assessment' path) to ensure tests are context-aware.
- Judgement Generation: Using the Granularity Principle, it generates specific yes/no questions for the LLM judge covering loading, workflow, and quality.
- User Review: It presents a categorized table of proposed judgements for the developer to approve or refine.
- YAML Serialization: Once approved, it either generates a new YAML config or updates the judge_definitions section of an existing one while preserving other metadata.
Generate Judgements for Skill Evaluation Setup
To use this tool within your development environment, ensure you have access to the skill directory. Trigger the workflow by calling the skill name:
# Example trigger phrases
"generate judgements for the current skill"
"create judge definitions from SKILL.md"
Ensure your project contains a references/yaml-config-spec.md if you need to reference specific configuration standards.
Generate Judgements for Skill Evaluation Data Schema & Taxonomy
The skill organizes its output into structured YAML configurations compatible with the mlflow-skills framework.
| Field | Description |
|---|---|
| name | Unique kebab-case identifier for the check (e.g., sequential-mcp-calls). |
| scope | Defined as 'all' or a specific execution path (e.g., 'checklist'). |
| question | A self-contained prompt instructing the LLM judge on what to verify in the trace. |
Files are typically saved in tests/configs/{skill-name}.yaml.
Generate Judgements for Skill Evaluation Advanced Features
- Scope-Aware Testing: Automatically segregates tests into 'all' for shared behavior and specific scopes for conditional logic.
- Negative Judgement Generation: Automatically creates checks to ensure artifacts from one scope do not leak into another.
- Incremental Updates: Intelligently updates existing YAML files by replacing only the judge_definitions while preserving prompts and environment variables.
- Trace-Level Specificity: Generates instructions specifically designed for LLMs to read raw execution traces (JSON tool calls/responses).
SKILL.md
Loading
Related Openclaw Skills
SCP Paradigm Analysis
A foundational economic framework used to analyze how industry structure dictates firm behavior and market performance.
American Express (Amex) Financial Intelligence
A high-level financial intelligence skill for analyzing American Express's unique closed-loop payment ecosystem and premium market positioning.
Household Renewal Command Center
A comprehensive organizational framework designed to track and manage all recurring household document renewals and administrative deadlines.
Movavi Video Editor
A cloud-based AI video editing skill that transforms raw footage into polished MP4 videos through simple natural language descriptions.
Hanoi Vietnam Knowledge Skill
A comprehensive intelligence profile of Hanoi, Vietnam, detailing its 1,000-year history, economic structure, and strategic importance in Southeast Asia.
Rust Engineer
A specialized AI agent skill for senior Rust development focusing on systems programming, zero-cost abstractions, and memory safety.
Featured*
