RAG Groundedness Metric Evaluation System using LLMs - n8n Workflow

RAG engineers and AI researchers needing to quantify model performance.
MLOps specialists who require automated quality assurance for LLM applications.
n8n developers building scalable and verifiable AI solutions.
Anyone focused on reducing LLM hallucinations by ensuring responses are faithful to source documents.

Developing a robust RAG system requires constant validation to ensure the generated responses are grounded in the provided context rather than hallucinated or derived from external knowledge. This comprehensive n8n workflow provides a solution by implementing an automated groundedness metric evaluation.

It first sets up a vector store (using the Bitcoin Whitepaper as an example document) and configures an n8n AI agent to answer questions using that store. The evaluation phase uses a dedicated n8n trigger to pull test data (questions) from a Google Sheet. A second, powerful OpenAI model then acts as an impartial evaluator, scoring the AI agent's response based on a strict rubric to determine if every part of the answer is attributable only to the retrieved documents. This approach allows developers using n8n to generate crucial, quantifiable quality metrics for their AI systems.

This automation operates in two distinct phases: RAG Store Preparation and Automated Evaluation.

1. RAG Preparation (Manual Trigger): The n8n workflow starts by fetching a document (the Bitcoin Whitepaper) via an HTTP Request. This document is then processed by a series of LangChain n8n node components (Text Splitter, OpenAI Embeddings) and inserted into a Simple Vector Store (in-memory knowledge base).


2. Evaluation Initiation (Evaluation Trigger): The evaluation process begins when the specialized When fetching a dataset row n8n trigger pulls a test query (input) from the configured Google Sheet evaluation dataset.


3. AI Agent Execution: The input query is fed to the AI Agent n8n node. This agent, configured with an OpenAI Chat Model (gpt-4o-mini), uses the RAG store as a retrieval tool (bitcoin_whitepaper) to generate a fully grounded answer.


4. Data Extraction: The Get Documents n8n node captures both the AI Agent's final response and the crucial intermediateSteps, specifically extracting the context documents retrieved by the RAG tool.


5. Grounding Assessment: The core evaluation occurs in the Document Grounding n8n node. This LangChain Chain LLM node sends the retrieved documents and the AI response to a scoring OpenAI model (gpt-4.1-mini). The model uses a defined prompt and a strict rubric (0 or 1) to assess if the response is fully grounded in the provided documents.


6. Reporting: The final rating and reason are structured via the Structured Output Parser n8n node. Finally, the Set Outputs n8n node writes the evaluation metrics (score, reason, output, documents used) back to the Google Sheet, completing the loop of the n8n workflow.

To deploy this n8n templates successfully, follow these steps:

1. Import the JSON: Copy the provided n8n workflow JSON and paste it into your n8n instance.


2. OpenAI Credentials: You must configure credentials for the Embeddings OpenAI nodes, the OpenAI Chat Model nodes, and the scoring model. Ensure you select the correct OpenAI credential for all these n8n node instances.


3. Google Sheets Credentials: Configure the Google Sheets OAuth2 API credentials for the When fetching a dataset row n8n trigger and the Evaluation nodes (Set Outputs, Set Metrics).


4. Initial RAG Setup: Execute the first branch (When clicking ‘Execute workflow’ down to Simple Vector Store) to load the Bitcoin Whitepaper into memory. This step is necessary to prepare the knowledge base for the AI Agent.


5. Evaluation Setup: Ensure the When fetching a dataset row n8n trigger is correctly configured to point to your evaluation Google Sheet document ID and sheet name. This n8n trigger is what initiates the evaluation runs.

When fetching a dataset row (n8n trigger): A specialized evaluation n8n trigger that connects to a Google Sheet. It pulls input data (test questions) row by row to initiate the RAG evaluation process.
Get Datasheet (HTTP Request n8n node): Fetches the source document (bitcoin.pdf) which will serve as the knowledge base for grounding.
Simple Vector Store: This LangChain n8n node acts as the in-memory RAG storage, indexed by OpenAI embeddings for fast retrieval by the AI Agent.
AI Agent (LangChain Agent n8n node): The core RAG application logic. It uses the OpenAI Chat Model (GPT-4o-mini) and is configured to use the vector store as a required tool to ground its answers.
Get Documents (Set n8n node): This crucial n8n node extracts the AI Agent's response (output) and the specific source documents retrieved during the process from the Agent's intermediateSteps data structure.
Document Grounding (LangChain Chain LLM n8n node): The scoring engine. It uses a separate OpenAI model (GPT-4.1-mini) and a detailed instruction prompt to perform the groundedness evaluation. It requires the previous retrieved documents and the agent's output as context for scoring.
Structured Output Parser: Ensures the evaluation score (rating 0 or 1) and rationale (reason) from the LLM are returned in a clean, parsable JSON format.
Set Outputs / Set Metrics (Evaluation n8n node): These nodes finalize the evaluation, pushing the generated outputs, scores, and metrics back into the Google Sheet for reporting and analysis.