An automated LLM-as-a-Judge evaluation system for scoring AI response quality, accuracy, and hallucination using Langfuse.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install llm-evaluator
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install llm-evaluator using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is LLM Evaluator?
LLM Evaluator is a professional-grade evaluation framework designed to bring rigors to AI development. By implementing the LLM-as-a-Judge pattern, it allows developers to programmatically score AI outputs on critical metrics like relevance and factual correctness. As a key component of the Openclaw Skills ecosystem, it leverages Langfuse for trace management and GPT-5-nano via OpenRouter for cost-effective, high-speed judging.
This skill bridges the gap between raw model outputs and actionable quality metrics. It provides a systematic way to monitor performance regressions, detect hallucinations in RAG pipelines, and ensure that agentic workflows meet production standards without requiring manual human intervention for every trace.
LLM Evaluator Use Cases
- Detecting hallucinations in RAG-based search results and AI responses.
- Automated quality assurance for complex agentic workflows.
- Backfilling evaluation scores for historical traces to identify long-term performance trends.
- Benchmarking different model versions based on empirical accuracy and helpfulness scores.
How LLM Evaluator Works
- Connect the skill to a Langfuse instance where your AI agent traces are stored.
- Configure the required environment variables, including the OpenRouter API key for the judge model.
- Execute the evaluation script to pull recent traces or target a specific trace ID.
- The system analyzes the input/output pair using GPT-5-nano against defined rubrics (relevance, accuracy, etc.).
- Scores are pushed back to Langfuse, appearing as tags or metrics on the original trace for easy visualization.
LLM Evaluator Setup
To deploy this evaluator within your Openclaw Skills environment, ensure you have Python 3.10+ installed and your Langfuse credentials ready.
# Configure the judge model provider
export OPENROUTER_API_KEY="your_openrouter_key"
# Verify the setup with sample cases
python3 scripts/evaluator.py test
# Score a specific trace for quality
python3 scripts/evaluator.py score <trace_id>
LLM Evaluator Data Schema & Taxonomy
The skill organizes its evaluation data using a standardized scoring system visible within your observability platform:
| Metric | Definition | Scale |
|---|---|---|
| relevance | How well the response aligns with the user's intent | 0 to 1 |
| accuracy | The factual correctness of the generated content | 0 to 1 |
| hallucination | Detection of made-up or unsupported information | 0 to 1 |
| helpfulness | The overall utility and clarity of the response | 0 to 1 |
LLM Evaluator Advanced Features
- Batch backfill capability to process up to 20 unscored traces in a single command.
- Granular evaluator selection to focus on specific metrics like relevance while skipping others to save tokens.
- Optimized for GPT-5-nano to provide state-of-the-art judging logic at a fraction of the cost of larger models.
- Seamless integration with the broader suite of Openclaw Skills for end-to-end agent monitoring and optimization.
SKILL.md
Loading
Related Openclaw Skills
LLM Evaluator
An automated LLM-as-a-Judge system that scores Langfuse traces for quality, accuracy, and hallucination detection.
LiveKit Voice
Manage real-time voice and video infrastructure for AI agents using LiveKit WebRTC.
Linode/Akamai Management Skill
A comprehensive Python-based CLI tool for managing Linode (Akamai) cloud infrastructure, including compute, networking, and Kubernetes.
Lemon Squeezy
A comprehensive CLI tool for managing Lemon Squeezy digital products, orders, and subscriptions within Openclaw Skills.
Loom
A Python-powered interface to manage Loom video recordings, transcripts, and workspace folders via the official Developer API.
Mac Mini AI Server for OpenClaw
A comprehensive deployment guide for running Openclaw Skills on a dedicated Mac Mini server for 24/7 AI availability.
Featured*
