A developer-focused CLI utility to benchmark AI API models over time, detecting quality degradation with local test runs and performance alerts.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install model-watch
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install model-watch using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is model-watch?
The model-watch tool is a standardized, local benchmarking suite built to monitor and detect quality degradation in hosted AI models. As AI providers frequently update their models, developers often complain about silent degradation and increased hallucinations. Rather than relying on subjective vibes or forum rumors, model-watch provides hard, reproducible metrics using local evaluations.
By utilizing model-watch through Openclaw Skills, developers can systematically protect their production workflows from sudden drops in model capability. It enables automatic tracking of historical performance across reasoning, coding, writing, and instruction-following, alerting you immediately if a model's capabilities deteriorate over time.
model-watch Use Cases
- Detecting silent AI model degradation: Prevent paying the same premium for underperforming, updated LLMs.
- Verifying API consistency: Integrate model-watch with Openclaw Skills workflows to regularly test model capabilities in development environments.
- Evaluating new model versions: Determine if newly released hosted model endpoints outperform your historical baseline.
- Regression testing: Catch drop-offs in complex coding and logical deduction tasks after upstream API changes.
How model-watch Works
- Generate Questions: Run the 7 standard benchmark questions through your selected target AI model.
- Submit Responses: Pipe or input the model's generated answers directly using the CLI command.
- Local Storage: The system parses the responses and stores calculated scores inside
~/.hermes/model-watch-history.json. - Track Performance Trends: Query historical performance and visualize rating trajectories using CLI tools.
- Trigger Alerts: Automatically evaluate if recent scores have dropped by more than 10% compared to your historical baseline to prevent application failures.
model-watch Setup
Install the model-watch skill directly via git:
pip install git+https://github.com/minirr890112-byte/model-watch.git
Verify the installation by running the following CLI commands:
# View the default benchmark questions to prompt your model
model-watch demo
# Submit answers from your AI model to calculate and save local scores
model-watch submit '{"reasoning_1":"...","coding_1":"...",...}'
# Review historical performance trends
model-watch history
# Check for silent degradation and trigger alerts
model-watch alert
model-watch Data Schema & Taxonomy
All performance history is organized and persisted locally to maintain privacy and allow custom analytics.
File Structure
- Location:
~/.hermes/model-watch-history.json - Format: JSON Array containing historical evaluation records.
Evaluation Schema Fields
| Category | Tests | Description |
|---|---|---|
| Reasoning | 2 tests | Measures logic, multi-step deduction, and mathematical sanity |
| Coding | 2 tests | Evaluates code generation, syntax correctness, and debugging |
| Writing | 1 test | Measures general coherence, quality, and style adherence |
| Instruction-following | 1 test | Evaluates precision and adherence to specified constraints |
| Hallucination detection | 2 tests | Measures factual accuracy and resistance to hallucinating false details |
model-watch Advanced Features
- Local Score Retention: Keeps your benchmark history completely secure, free from external SaaS tracking.
- Degradation Detection Engine: Automatically flags performance drops of greater than 10% compared to your custom historical average.
- Standardized Multi-Category Benchmarks: Evaluates models across 5 critical real-world domains rather than relying on synthetic laboratory metrics.
- Automated Workflow Integration: Easily trigger model-watch alert runs inside CI/CD pipelines or nightly cron tasks via Openclaw Skills configurations.
SKILL.md
Loading
Related Openclaw Skills
llm-deploy-helper
A local hardware detection CLI that automatically analyzes system memory, GPU, and CPU specs to suggest and output ideal Ollama and llama.cpp setup commands.
Wall-Mounted Faucet Layout Calculator
A high-precision mathematical calculator and SVG visualizer to resolve wall-mounted faucet reach, outlet height, and drain alignment coordinates.
Tobacco Network Academy Auto Learning Skill (cttc-auto-learn)
An automated browser-automation agent designed to manage course enrollment, video playback, and study hour tracking on the China Tobacco Network Academy platform.
Contract Review Assistant
An AI-powered local contract review agent that generates multidimensional risk heatmaps, interactive HTML reports, and tailored negotiation strategies.
api-cost-compare
A command-line tool to compare pricing across 18 AI models from 6 providers, recommend the cheapest options for specific use cases, and track API spend.
Code Inspector
A command-line tool that performs AST-based static analysis on AI-generated Python code to detect bugs, security flaws, and assess production-readiness.
Featured*
