An automated LLM-as-a-Judge system that scores Langfuse traces for quality, accuracy, and hallucination detection.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install llm-evaluator-pro
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install llm-evaluator-pro using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is LLM Evaluator?
The LLM Evaluator is a sophisticated quality assurance tool designed for developers building agentic workflows. By leveraging Langfuse and LLM-as-a-Judge methodologies, it provides objective scoring for AI outputs based on critical metrics like relevance and helpfulness. This skill is a vital component of the Openclaw Skills ecosystem, enabling teams to monitor performance and maintain high standards for their AI agents through automated observability and robust scoring mechanisms.
LLM Evaluator Use Cases
- Quantifying the relevance and accuracy of AI search results or chat responses.
- Detecting hallucinations in generated content before it reaches end-users.
- Performing batch backfills of scores on legacy traces for historical performance analysis.
- Integrating automated quality assurance into the agent development lifecycle.
How LLM Evaluator Works
- The skill interfaces with Langfuse to retrieve specific traces or batches of unscored logs.
- It utilizes a high-performance judge model via OpenRouter to analyze the prompt-response pair.
- The evaluator applies predefined logic to score the trace across four dimensions: relevance, accuracy, hallucination, and helpfulness.
- Results are pushed back to the Langfuse dashboard, providing visual feedback and data for analytics.
LLM Evaluator Setup
To get started with this skill in the Openclaw Skills environment, ensure you have your environment variables configured and Python 3 installed.
# Set required API keys
export OPENROUTER_API_KEY="your_key"
export LANGFUSE_PUBLIC_KEY="your_key"
export LANGFUSE_SECRET_KEY="your_key"
# Run a test to verify configuration
python3 scripts/evaluator.py test
LLM Evaluator Data Schema & Taxonomy
The evaluator organizes data based on standard observability metrics. Scores are returned on a scale of 0 to 1.
| Evaluator | Measures | Scale |
|---|---|---|
| relevance | Response relevance to query | 0–1 |
| accuracy | Factual correctness | 0–1 |
| hallucination | Made-up information detection | 0–1 |
| helpfulness | Overall usefulness | 0–1 |
LLM Evaluator Advanced Features
- Selective Evaluation: Use the --evaluators flag to trigger specific scoring metrics only.
- Batch Backfilling: Efficiently process large volumes of unscored historical traces using the backfill command.
- Test Mode: Validate your judge model configuration with sample cases before running on production data.
- AgxntSix Integration: Optimized for the AgxntSix suite within the Openclaw Skills framework.
SKILL.md
Loading
Related Openclaw Skills
LiveKit Voice
Manage real-time voice and video infrastructure for AI agents using LiveKit WebRTC.
Linode/Akamai Management Skill
A comprehensive Python-based CLI tool for managing Linode (Akamai) cloud infrastructure, including compute, networking, and Kubernetes.
Lemon Squeezy
A comprehensive CLI tool for managing Lemon Squeezy digital products, orders, and subscriptions within Openclaw Skills.
Lead Gen Pipeline
An AI-powered automation skill for scoring leads and generating context-aware sales follow-ups.
LLM Evaluator
An automated LLM-as-a-Judge evaluation system for scoring AI response quality, accuracy, and hallucination using Langfuse.
Loom
A Python-powered interface to manage Loom video recordings, transcripts, and workspace folders via the official Developer API.
Featured*
