A professional-grade auditing tool designed to measure AI agent intelligence through 14 distinct dimensions including planning, hallucination control, and trend analysis.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install smartness-eval-open-source
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install smartness-eval-open-source using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is OpenClaw Smartness Eval?
OpenClaw Smartness Eval is a sophisticated evaluation framework designed to move beyond anecdotal evidence of AI performance. Instead of judging an agent based on a single impressive response, this skill provides a structured, multi-dimensional audit of an AI's capabilities. It aligns with industry standards such as CLEAR, T-Eval, and Anthropic benchmarks to ensure that Openclaw Skills remain robust, reliable, and truly intelligent over time.
The tool analyzes real-world execution logs, latency metrics, and reasoning depth to produce a comprehensive score. It is particularly valuable for developers who need to verify if a version upgrade actually improved the model's logic or if there has been a regression in specific cognitive dimensions.
OpenClaw Smartness Eval Use Cases
- Performing regression testing after AI model or system updates to confirm capability gains.
- Generating structured weekly or monthly self-evaluation reports for AI performance auditing.
- Identifying specific dimensions of performance degradation to target technical debt.
- Preparing standardized capability assessment results for external demonstrations or stakeholders.
How OpenClaw Smartness Eval Works
- The skill aggregates data from multiple sources including response latency metrics, error trackers, and reasoning stores.
- It executes a suite of validated test commands defined in the task-suite configuration.
- Logic engines apply standardized rubrics across 14 dimensions like planning, evidence-based reasoning, and risk management.
- Optional LLM-based judging (via DeepSeek or OpenAI) is triggered for subjective scoring if requested.
- The system generates a comprehensive JSON data file and a human-readable Markdown report comparing results against historical trends.
OpenClaw Smartness Eval Setup
To install and initialize the evaluation skill, ensure your environment has access to the standard Python library. For advanced LLM judging, configure your API keys first:
# Set API keys for optional LLM judging
export DEEPSEEK_API_KEY='your_key_here'
# Run a standard evaluation
python3 skills/openclaw-smartness-eval/scripts/eval.py --mode standard
# Run a quick check to verify installation and health
python3 skills/openclaw-smartness-eval/scripts/check.py
OpenClaw Smartness Eval Data Schema & Taxonomy
The skill maintains a strict data hierarchy within the state/smartness-eval/ directory to ensure historical tracking of Openclaw Skills performance:
| File Type | Path | Purpose |
|---|---|---|
| Run Data | runs/<timestamp>.json |
Complete raw evaluation data for a single execution. |
| Reports | reports/<date>.md |
Human-readable summaries and scorecards. |
| History | history.jsonl |
Append-only log of scores for longitudinal trend analysis. |
Key metrics captured include overall_score, dimension_spread, risk_flags, and pass_at_k reliability scores.
OpenClaw Smartness Eval Advanced Features
- Deep Mode Analysis: Executes repetitive test runs with a 30-day window for high-confidence pass@k metrics.
- LLM Judge Integration: Leverages external LLMs to provide qualitative feedback and subjective scoring rubrics.
- Degradation Alerts: Automatically flags any capability dimension that drops by more than 5 points compared to the last run.
- Secure Command Validation: A whitelist-based execution engine that prevents unsafe shell or python injections during testing.
SKILL.md
Loading
Related Openclaw Skills
Amap Maps Streamable HTTP Skill
A robust set of mapping tools providing AI agents with professional-grade routing, geocoding, weather data, and location-based search capabilities.
Finance Reporter
A real-time financial data delivery tool that tracks global stocks, forex, commodities, and crypto via the Yahoo Finance API.
Alaska Airlines Scraper
A high-performance scraping tool for fetching Alaska Airlines award flight data and monthly mileage calendars.
Trending Topic Finder
An automated utility designed to fetch raw viral topic recommendation data directly from backend service interfaces.
Job Search Agent
An AI-powered job search assistant that automates multi-platform searching, CV matching, and the application process.
ClawMover Migration
A security-first migration skill designed to guide users through safe OpenClaw backup, restore, and environment synchronization workflows.
Featured*
