RAG Document Relevance Evaluation Metric Calculation - n8n Workflow

• AI/ML Engineers and Data Scientists needing automated RAG system evaluation.
• Developers utilizing n8n templates for advanced AI and Large Language Model (LLM) automation.
• Users seeking to implement custom, prompt-based metrics using an n8n node structure.
• Anyone managing RAG systems who needs to systematically test knowledge base retrieval quality.

Evaluating the quality of a Retrieval-Augmented Generation (RAG) system is critical for production readiness. This sophisticated n8n workflow calculates the 'retrieved document relevance' metric. This metric determines if the information (facts) retrieved from the vector store by the AI Agent is actually relevant to the user's initial question.

This specific n8n workflow is divided into two parts: a setup flow to populate the vector store from a Google Sheet dataset, and the main evaluation flow. By using this n8n template, you gain a repeatable, transparent method for assessing retrieval performance, essential for debugging and improving your RAG application's knowledge foundation. The conditional logic ensures that the metric calculation (the most costly step) only runs when the n8n trigger is executing a formal evaluation.

The n8n workflow operates in two main phases:

Phase 1: Knowledge Base Setup

1. The setup phase is triggered manually via the 'When clicking ‘Execute workflow’' n8n node.


2. The 'Get dataset' Google Sheets n8n node retrieves raw documents from the configured spreadsheet.


3. Documents are cleaned using 'Remove Duplicates'.


4. The content is processed by the 'Recursive Character Text Splitter' and loaded via the 'Default Data Loader'.


5. 'Embeddings OpenAI' generates vectors, which are then inserted into the 'Simple Vector Store' (in-memory knowledge base).

Phase 2: Metric Calculation

1. The main flow begins with one of two n8n trigger nodes: either 'When fetching a dataset row' (for bulk evaluation) or 'When chat message received' (for live testing).


2. If triggered by the dataset, 'Match chat format' prepares the input for the agent.


3. The 'AI Agent' n8n node processes the question, retrieves relevant documents from the knowledge base (using 'Simple Vector Store1'), and formulates a response. Crucially, it returns 'intermediate steps' detailing its tool usage.


4. The 'Evaluating?' n8n node checks if a formal evaluation is running. If yes, it proceeds to metric calculation; otherwise, it stops the metric process to save costs.


5. 'Extract documents' pulls the raw text of the retrieved documents from the agent's intermediate steps.


6. The 'Calculate doc relevance metric' OpenAI n8n node receives the user's QUESTION and the retrieved FACTS. It uses a rigorous system prompt to score the relevance (0 or 1) and provides detailed reasoning, outputting structured JSON.


7. Finally, the 'Set metrics' Evaluation n8n node records the calculated score (similarity) into the evaluation run data, completing this powerful n8n workflow step.

1. Import the n8n workflow: Copy the provided JSON and paste it into your n8n instance's canvas via the 'Import from JSON' option.


2. Google Sheets Credentials: Configure the 'Get dataset' and 'When fetching a dataset row' n8n trigger nodes with valid Google Sheets credentials linked to the source dataset URL.


3. OpenAI Credentials: Ensure that the 'Embeddings OpenAI', 'OpenAI Chat Model', and 'Calculate doc relevance metric' n8n node instances are configured with your valid OpenAI API key credentials.


4. Setup Run: Before running the main workflow, execute the 'Setup: Populate vector DB' path (starting with 'When clicking ‘Execute workflow’') once to load the documents into the in-memory vector store.


5. Execution: Run the main flow via the 'When fetching a dataset row' n8n trigger to start the batch evaluation.

When fetching a dataset row (n8n trigger): This Evaluation n8n node initiates the relevance check for each row of the test dataset, providing the required inputs (like the question) to the RAG system.
AI Agent (Langchain n8n node): Serves as the core RAG processor. It uses the 'OpenAI Chat Model' and the 'Simple Vector Store1' tool to retrieve information. Key configuration: 'Return intermediate steps' is enabled to capture the retrieved documents.
Evaluating? (Evaluation n8n node): A critical flow control n8n node. It ensures that the subsequent high-cost metric calculation is only executed during a formal evaluation run, saving operational costs.
Extract documents (Set n8n node): Used to parse the output from the AI Agent, specifically filtering the intermediate steps to isolate the observation from the vectorknowledgebase tool, which contains the retrieved FACTS.
Calculate doc relevance metric (OpenAI n8n node): This n8n node performs the actual metric calculation. It uses a GPT-4o-mini model with a detailed system prompt to critically assess the relevance of the retrieved documents against the original question, returning a score (0 or 1) and comprehensive reasoning in JSON format.
Set metrics (Evaluation n8n node): The final n8n node in the evaluation chain, used to formally record the calculated relevance score (named similarity) back into the n8n evaluation result set.