A diagnostic tool for evaluating the accuracy and retrieval quality of RAG pipelines using Ragas metrics.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install rag-eval
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install rag-eval using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is RAG Eval?
RAG Eval is a sophisticated diagnostic utility designed to measure the performance and reliability of Retrieval-Augmented Generation (RAG) systems. As a key component in the Openclaw Skills ecosystem, it allows developers to quantify how well their AI agents retrieve information and generate contextually accurate responses. By utilizing the Ragas framework, the skill provides objective scoring for faithfulness, answer relevancy, and context precision.
This skill is essential for developers who need to ensure their AI applications remain production-ready and free from hallucinations. It acts as an automated judge, bridging the gap between raw LLM outputs and verifiable data quality. By integrating RAG Eval into your workflow, you can maintain high standards for your Openclaw Skills by continuously monitoring the effectiveness of your vector databases and retrieval logic.
RAG Eval Use Cases
- Quantifying the accuracy of AI agent responses against retrieved source documents.
- Detecting hallucinations in RAG pipelines by calculating faithfulness scores.
- Optimizing vector database retrieval by evaluating context precision and relevancy.
- Benchmarking different LLMs (OpenAI, Anthropic, or local models) as evaluation judges.
- Performing automated batch quality testing on large datasets for regression analysis.
How RAG Eval Works
- The system collects the original user question, the generated answer, and the retrieved context chunks.
- Input data is securely written to a temporary JSON file to prevent command injection risks.
- A Python-based evaluation engine processes the data using the Ragas library and the configured LLM judge.
- Specific metrics including faithfulness and answer relevancy are calculated for the provided interaction.
- The skill generates a JSON output containing individual metric scores and a final PASS/FAIL verdict.
- Results are saved to a local results directory for historical tracking and performance analysis within Openclaw Skills.
RAG Eval Setup
To get started with RAG Eval, you can install it directly through the CLI or via your agent.
# Install the skill via the hub
clawhub install rag-eval
# Run the setup script to install Ragas and Python dependencies
bash scripts/setup.sh
You must also configure at least one LLM API key to act as the evaluation judge:
export OPENAI_API_KEY="your_openai_key"
# Or for local evaluation
export RAGAS_LLM="ollama/llama3"
RAG Eval Data Schema & Taxonomy
The skill organizes its evaluation data and metadata to provide clear insights into pipeline performance.
| Metric | Description | Target Score |
|---|---|---|
| faithfulness | Accuracy of the answer based on provided context | > 0.85 |
| answer_relevancy | How well the answer addresses the user's question | > 0.85 |
| context_precision | The quality and ranking of retrieved document chunks | > 0.80 |
All evaluation logs are stored in memory/eval-results/ using date-stamped JSONL files, allowing for easy integration with other Openclaw Skills for data visualization.
RAG Eval Advanced Features
- Batch Evaluation: Process entire JSONL datasets in a single command to evaluate system-wide performance.
- Deep-Dive Explanations: Use the
--explainflag to see exactly which parts of an answer are considered unsupported by the context. - Local LLM Support: Run evaluations entirely offline using Ollama or other local inference engines to reduce costs and maintain privacy.
- Customizable Thresholds: Fine-tune the PASS and REVIEW score requirements via environment variables to suit specific project needs.
- Multi-Agent Compatibility: Easily integrate evaluation steps into complex multi-agent workflows using Openclaw Skills.
SKILL.md
Loading
Related Openclaw Skills
OpenClawBrain
A self-regulating, learned memory graph that enables AI agents to evolve through autonomous feedback and policy-gradient routing.
Ahrefs MCP
A professional-grade SEO integration that connects Ahrefs data to your AI agent via the Model Context Protocol.
Ahrefs SEO Connector
A comprehensive SEO toolkit for AI agents to access real-time Ahrefs data for keywords, backlinks, and site audits.
TestFlight Seat Monitor
A smart monitoring tool that tracks TestFlight beta availability and alerts you only when new slots become available.
Research Library
A local-first multimedia research library designed to capture, organize, and search hardware project knowledge with intelligent weighting.
Decision Topology
A local-first skill that records the structural evolution of ideas, branches, and pivots in your conversations as a browsable JSON tree.
METADATA
Featured*
