A comprehensive Python SDK for evaluating generative AI application quality, safety, and performance through built-in and custom metrics.
Install & Download
1. ClawHub CLI
The fastest way to install a skill directly from the registry.
npx clawhub@latest install azure-ai-evaluation-py
2. Manual Installation
Copy the skill folder to one of these locations
~/.openclaw/skills/ <project>/skills/ Priority: Workspace > Local > Bundled
3. Prompt Installation
Copy this prompt to OpenClaw to install it automatically.
Help me install azure-ai-evaluation-py using Clawhub. If Clawhub is not installed, install it first (npm i -g clawhub).
Prefer to download?
Get the raw skill files in a ZIP archive.
What is Azure AI Evaluation (Python)?
The Azure AI Evaluation skill enables developers to systematically assess the performance of generative AI applications using the official Python SDK. It provides a suite of AI-assisted evaluators to measure complex traits like groundedness and coherence, alongside traditional NLP-based metrics such as F1 and ROUGE scores. By leveraging this tool within your Openclaw Skills collection, you can ensure that LLM-driven outputs meet strict quality and safety standards before reaching end-users.
Beyond simple scoring, this skill facilitates deep integration with Azure AI Foundry, allowing for centralized logging and visualization of evaluation runs. Whether you are building RAG systems or conversational agents, this framework provides the analytical rigor needed to iterate on prompts and model configurations with confidence.
Azure AI Evaluation (Python) Use Cases
- Validating RAG pipelines by checking the groundedness of responses against retrieved context.
- Screening AI outputs for safety violations including hate speech, violence, and self-harm.
- Comparing multiple model deployments or prompt versions through batch evaluation runs.
- Automating quality assurance within CI/CD pipelines for AI agents.
- Developing custom, domain-specific evaluation logic for specialized industry requirements.
How Azure AI Evaluation (Python) Works
- Configure the environment with necessary Azure OpenAI endpoints and API credentials.
- Initialize specific evaluators, such as GroundednessEvaluator or RelevanceEvaluator, with the appropriate model configuration.
- Prepare evaluation data in formats like JSONL, ensuring query, context, and response fields are present.
- Execute the evaluation using either single-row logic or the batch evaluate() function.
- Map data columns to evaluator inputs to ensure accurate metric calculation.
- Review generated scores and qualitative reasoning or export results to Azure AI Foundry for further analysis.
Azure AI Evaluation (Python) Setup
Install the primary SDK and its remote extension via pip:
pip install azure-ai-evaluation
pip install azure-ai-evaluation[remote]
Set up the required environment variables for the evaluation models:
export AZURE_OPENAI_ENDPOINT="https://<resource>.openai.azure.com"
export AZURE_OPENAI_API_KEY="<your-api-key>"
export AZURE_OPENAI_DEPLOYMENT="gpt-4o-mini"
Azure AI Evaluation (Python) Data Schema & Taxonomy
The skill processes structured data for evaluation and produces detailed metric reports. The following table describes the primary data components:
| Component | Format | Description |
|---|---|---|
| Input Data | JSONL | Must contain keys for query, context, and response for AI-assisted evaluators. |
| Quality Metrics | Float (1-5) | Scores for groundedness, relevance, coherence, fluency, and similarity. |
| Safety Metrics | Integer (0-7) | Severity ratings for violence, sexual content, and self-harm. |
| NLP Metrics | Float (0-1) | Traditional scores like F1, BLEU, and ROUGE. |
| Metadata | Object | Includes reasoning strings and Azure AI Foundry project links. |
Azure AI Evaluation (Python) Advanced Features
- Composite evaluators like QAEvaluator and ContentSafetyEvaluator for streamlined, multi-metric assessments.
- Support for custom prompt-based evaluators using the PromptChatTarget class for unique brand voice or logic checks.
- Direct evaluation of application targets (callables), enabling end-to-end testing of live application logic.
- Integration with Azure AI Project scope for enterprise-grade tracking and cross-team comparison of evaluation results.
- Flexible column mapping to handle diverse data schemas without manual preprocessing.
SKILL.md
Loading
Related Openclaw Skills
Azure AI Agents Python SDK
A low-level Python SDK for building, managing, and deploying intelligent AI agents hosted on Azure AI Foundry.
Azure Developer CLI (azd) Container Apps Deployment
A professional-grade automation skill for deploying multi-service containerized applications to Azure using the Azure Developer CLI (azd) and Bicep.
Agent Framework Azure Hosted Agents
A comprehensive Python framework for deploying persistent AI agents on Azure AI Foundry with native support for hosted tools and MCP servers.
Reddit Integration Skill
A comprehensive integration for Openclaw Skills that enables automated browsing, posting, and moderation of Reddit communities.
Azure AI Projects Python SDK
A high-level Python SDK for building, managing, and evaluating enterprise AI applications within the Azure AI Foundry ecosystem.
Azure AI Transcription Python
A powerful Python SDK for integrating Azure AI speech-to-text capabilities, supporting both batch and real-time transcription.
Featured*
