AI Agent Response Quality Assessment (RAGAS Correctness) - n8n Workflow

AI Developers and ML Engineers needing automated performance testing.
Users building Retrieval-Augmented Generation (RAG) systems.
Technical professionals requiring quantifiable quality metrics for their n8n workflow outputs.
Anyone seeking advanced LLM evaluation features within n8n templates.

Evaluating the output of AI agents and large language models (LLMs) is crucial but often complex. This advanced n8n workflow solves this by implementing the 'Correctness' metric, adapted from the RAGAS evaluation methodology. This n8n template provides a structured and quantifiable scoring mechanism by addressing two primary dimensions: factuality and semantic overlap. The logic of this n8n workflow uses an LLM (OpenAI) to classify response statements against the ground truth (True Positives, False Positives, False Negatives) to derive an F1 score, while simultaneously calculating the cosine similarity of the embeddings. The final correctness score, calculated within the n8n node, is a weighted average of these two metrics, giving you objective feedback on your AI agent's performance. Utilizing this powerful n8n workflow ensures your AI applications maintain accuracy and reliability.

This comprehensive n8n workflow starts with the dedicated When fetching a dataset row n8n trigger, designed specifically for evaluation runs. This n8n trigger pulls test data (question and ground truth) from a configured Google Sheet.

1. Agent Execution: The raw input is passed to the AI Agent n8n node, which uses an OpenAI Chat Model to generate the test answer.


2. Input Preparation: The Set Input Fields n8n node structures the question, answer, and groundTruth into a format suitable for two concurrent evaluation paths.


3. Path 1: F1 Score Calculation (Factuality): The Correctness Classifier n8n node uses a system prompt and the OpenAI model to classify individual statements in the answer against the ground truth (TP, FP, FN). The Calculate F1 Score n8n node then computes the F1 metric based on these counts, providing a precision-recall measure of correctness.


4. Path 2: Semantic Similarity: This parallel path uses the HTTP Request n8n node to interact with the OpenAI Embeddings API. It generates embeddings for both the agent's answer and each individual groundTruth statement. The Calculate Similarity Score n8n node then executes a custom JavaScript function to find the average cosine similarity between the answer and all ground truth vectors.


5. Final Scoring and Reporting: The scores from both paths are merged. The Correctness Score n8n node applies weighted averaging (75% F1 Score, 25% Similarity) to produce the final score for the evaluated item. Finally, the Update Outputs n8n node reports this definitive score back to the Google Sheets dataset, completing the robust evaluation cycle of this n8n workflow.

To deploy and utilize this powerful n8n workflow, follow these steps:

1. Import the n8n Workflow: Copy the provided JSON code and import it directly into your n8n instance.


2. OpenAI Credentials: Ensure you have configured an OpenAI API Key credential. You must link this credential to the OpenAI Chat Model, OpenAI Chat Model1, Get Embeddings, and Get Embeddings1 n8n node components.


3. Google Sheets Setup: Set up a Google Sheets credential (OAuth2) and link it to the When fetching a dataset row and Update Outputs n8n node components. The sheet must contain columns for 'input' (question), 'answer' (to be generated by the agent), and 'ground truth'.


4. Configure Evaluation Trigger: In the When fetching a dataset row n8n trigger node, select your Google Sheet and the relevant tab where your evaluation data resides.


5. Execution: Run the n8n workflow manually via the Evaluations interface. The specialized n8n trigger will iterate through your test dataset, evaluate each row, and update the score.

When fetching a dataset row (n8n trigger): A specialized n8n trigger that initiates the evaluation process by pulling input (question and ground truth) data from a configured Google Sheet.
AI Agent (Langchain Agent n8n node): Represents the system under test. It takes the input and generates the response (the 'answer') that will be evaluated for correctness.
Correctness Classifier (ChainLlm n8n node): This crucial n8n node uses a detailed prompt and the OpenAI model to analyze the agent's response against the ground truth, classifying statements as True Positive (TP), False Positive (FP), or False Negative (FN).
OpenAI Chat Model (LLM n8n node): Provides the large language model necessary for the classification task within the n8n workflow, ensuring accurate factuality assessment.
Get Embeddings/Get Embeddings1 (HTTP Request n8n node): Calls the OpenAI Embeddings API to generate vector representations of the agent's answer and the ground truth statements, required for the semantic similarity metric.
Calculate F1 Score (Code n8n node): Executes custom JavaScript code to calculate the F1 score based on the TP, FP, and FN counts provided by the classifier n8n node.
Calculate Similarity Score (Code n8n node): Executes custom JavaScript code to compute the average cosine similarity between the answer embedding and the ground truth embeddings.
Correctness Score (Code n8n node): Calculates the final weighted average score, combining the F1 score and the similarity score into a single, comprehensive correctness metric for the n8n workflow evaluation.

• Update Outputs (Evaluation n8n node): The final step of the n8n workflow, responsible for writing the calculated correctness score back to the original Google Sheet dataset for persistent metrics tracking.