AI Agent Accuracy Evaluation for Support Triage - n8n Workflow

AI/ML Engineers needing to test model performance in a live system.
Automation Specialists deploying AI Agents for customer support triage.
Users seeking advanced examples of n8n templates for evaluation metrics.
Anyone needing to validate the output of an OpenAI integration in an n8n node structure.

When deploying AI Agents for mission-critical tasks like support triage, validation is paramount. This n8n workflow addresses the challenge of verifying AI outputs by running a systematic evaluation against a known dataset (the 'ground truth'). This specific n8n template focuses on calculating a binary metric: did the AI Agent's classification (category and priority) match the expected answer? By using the specialized n8n evaluation nodes, this automation provides clear, quantifiable data on the agent's performance, enabling rapid iteration and refinement of the system prompt. This robust n8n workflow ensures your support automation is accurate and reliable before full deployment.

This automation is primarily designed to function within the n8n evaluation environment, though it can also be triggered via a standard webhook.

1. Trigger & Data Ingestion: The flow is initiated by the When fetching a dataset row (an n8n trigger for evaluation mode). This node pulls individual support ticket entries (subject, body, expected category, expected priority) from a specified Google Sheets dataset.


2. Input Formatting: The Match webhook format n8n node takes the data fields (subject and body) and transforms them into a raw JSON format expected by the downstream AI Agent, simulating a real-world incoming webhook request.


3. AI Processing: The AI Agent n8n node takes the formatted ticket data and, guided by a specialized system prompt, outputs a classified category and priority. This process uses the OpenAI Chat Model and a Structured Output Parser to ensure the output is always clean JSON, crucial for reliable metric comparison.


4. Evaluation Check: The flow passes into the Evaluating? n8n node, which ensures the metric calculation steps only run when the n8n workflow is in evaluation mode. If not evaluating, the flow returns a standard webhook response via Respond to Webhook.


5. Metric Calculation: The Check categorization n8n node uses custom expressions to perform the boolean comparison. It creates two new fields: categorymatch and prioritymatch. Each is set to true or false based on whether the AI output matches the expectedcategory or expectedpriority from the dataset row.


6. Setting Final Metrics: Finally, the Set metrics n8n node converts these boolean match results (true=1, false=0) into quantifiable metrics, which are then aggregated by the n8n evaluation system.

To install and utilize this n8n workflow, follow these steps:

1. Import: Copy the provided JSON into your n8n instance using the 'New' menu and selecting 'Import from JSON'. This n8n template requires specific configuration.


2. Credentials Setup:

1. Evaluation Configuration: Ensure the When fetching a dataset row n8n node is correctly configured with the URL of your evaluation Google Sheet, containing columns for subject, body, expectedcategory, and expectedpriority.


2. Testing: Save the n8n workflow. Run the evaluation by navigating to the workflow evaluation section in n8n and starting a new run. The system will automatically iterate through your dataset and calculate the accuracy metrics.

This n8n workflow relies on several specialized and core nodes:

When fetching a dataset row (Evaluation Trigger): Serves as the primary n8n trigger in evaluation mode. It reads row-by-row data from the Google Sheet URL provided, pulling in the subject, body, and ground truth data.
AI Agent (Langchain Integration): The core intelligence of this n8n node. It uses a structured prompt to enforce its role as a support triage assistant and define the required output format (category and priority JSON).
OpenAI Chat Model (GPT-4o-mini): Provides the language model backend for the AI Agent. Credentials must be linked to this n8n node.
Structured Output Parser (Langchain Integration): Ensures that the response from the OpenAI model conforms strictly to the expected JSON schema, preventing errors in the subsequent metric calculations.
Check categorization (Set): A critical n8n node that uses dynamic expressions (={{ $json.output.category == $('When fetching a dataset row').item.json.expected_category }}) to compare the AI's generated output against the ground truth data, resulting in boolean match flags.
Evaluating? (Evaluation): An n8n node that acts as a flow gate, checking the execution environment to ensure metric setting only occurs during a formal evaluation run.

• Set metrics (Evaluation): The final n8n node that registers the calculated accuracy scores (1 or 0) back into the n8n evaluation tracking system for reporting and aggregation.