Promptfoo is a powerful framework for running local, repeatable LLM evaluations and adversarial red-team security scans to ensure model reliability.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install promptfoo
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install promptfoo using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Promptfoo?
Promptfoo provides a robust developer workflow for testing LLM outputs against predefined requirements using a structured evaluation harness. It enables teams to move beyond manual prompt engineering by implementing automated regression suites, provider comparisons, and deep RAG evaluations. As part of the Openclaw Skills ecosystem, it ensures that AI applications remain secure, performant, and cost-effective through data-driven testing.
By utilizing a declarative configuration format, Promptfoo allows developers to define prompts, providers, and assertions in a single promptfooconfig.yaml. This approach facilitates systematic testing of model changes, ensuring that optimizations in one area do not cause regressions in another. Whether you are building a simple chatbot or a complex multi-agent system, Promptfoo provides the metrics and tooling necessary for production-grade AI development.
Promptfoo Use Cases
- Comparing performance and cost across multiple LLM providers like OpenAI, Anthropic, and local models.
- Running adversarial red-team scans to identify jailbreaks, prompt injections, and PII leaks.
- Evaluating RAG systems for context faithfulness, answer relevance, and factual accuracy.
- Automating regression testing for AI agents to ensure consistent tool-use and trajectory behavior.
- Generating synthetic datasets and assertions to expand test coverage quickly.
How Promptfoo Works
- Configuration: Define your evaluation logic in a
promptfooconfig.yamlfile, including prompts, target providers, and test variables. - Validation: Use the validate command to ensure your configuration matches the official schema and that all file references are intact.
- Execution: Run the evaluation engine which sends requests to the specified providers and collects responses based on your defined test cases.
- Assertion: Automatically grade the outputs using deterministic checks (regex, contains) or model-graded metrics (llm-rubric, factuality).
- Analysis: Review the results through a localized web interface or CLI logs to identify failures and performance bottlenecks.
- Iteration: Refine prompts or system logic and re-run the evaluation to confirm improvements.
Promptfoo Setup
To get started with Promptfoo within the Openclaw Skills environment, use the following commands:
# Initialize a new project with a template
npx promptfoo@latest init --example getting-started
# Validate your configuration file
npx promptfoo@latest validate
# Run the evaluation suite
npx promptfoo@latest eval
# View the results in your browser
npx promptfoo@latest view
Promptfoo Data Schema & Taxonomy
Promptfoo organizes evaluation data through a structured YAML configuration and local cache files. The primary components include:
| Component | Description |
|---|---|
| promptfooconfig.yaml | The main configuration file defining prompts, providers, and tests. |
| Prompts | Can be raw strings or external files referenced via file://. |
| Providers | Configuration for API endpoints (OpenAI, Anthropic) or local scripts. |
| Tests | Sets of variables and assertions used to validate model outputs. |
| Assertions | Logic-based or LLM-based metrics used to determine a Pass/Fail status. |
| Evals | Historical records of runs, accessible via promptfoo list evals. |
Promptfoo Advanced Features
- Red-Team Automation: Built-in specialized workflows for vulnerability scanning and policy enforcement.
- MCP Integration: Expose evaluation capabilities to any Model Context Protocol compatible toolchain.
- Agent Tracing: Support for evaluating complex agent workflows, including tool-calling and multi-turn interactions.
- CI/CD Integration: Seamlessly integrate LLM evals into your deployment pipeline to prevent regressive prompt changes.
- Custom Assertions: Extend the framework with specialized Python or JavaScript logic for domain-specific validation.
SKILL.md
Loading
Related Openclaw Skills
copypaste-cloud
A streamlined integration for interacting with the copy-paste.cloud pastebin to share code snippets and retrieve public text data.
Skill Auto Router
A deterministic utility for matching user tasks to the most relevant installed skills while maintaining library health through daily audits.
Playwright Pro
A comprehensive, production-grade toolkit for AI coding agents to architect, generate, and maintain robust Playwright test suites.
Hello World Skill
A foundational testing skill designed to demonstrate custom AI agent greeting protocols and basic skill integration.
Infoway Financial Data
A comprehensive MCP server providing real-time financial market data, sector analysis, and company fundamentals for over 20,000 instruments.
Accessibility Checker
A professional tool for auditing and improving web accessibility to ensure WCAG compliance and inclusive user experiences.
Featured*
