AI Summarization Quality Assessment using LLM Evaluation - n8n Workflow

• AI/ML Engineers testing Generative AI performance.


• Data Scientists requiring quantifiable metrics for LLM outputs.


• Developers utilizing n8n templates for advanced AI automation.


• Anyone needing robust evaluation metrics for summarization tasks within an n8n workflow.

The challenge in deploying production-ready AI models is ensuring consistent quality and mitigating issues like hallucination. This specialized n8n workflow provides a solution by implementing a secondary LLM judge to rigorously score the primary model's output against defined criteria. This particular n8n workflow focuses on Summarization Quality, measuring metrics like groundedness (ensuring the summary uses only source material), instruction following, conciseness, and fluency. By utilizing this advanced n8n node structure, developers gain reliable, quantitative feedback on their AI processes, making this one of the most valuable n8n templates for quality control in Generative AI applications.

1. Trigger: The process begins when the n8n trigger fires. This can be via an incoming webhook providing a Google Drive URL or by using the When fetching a dataset row n8n trigger to pull inputs from a Google Sheets evaluation dataset.


2. Transcript Retrieval: The Google Drive n8n node downloads the video transcript file using the provided URL. The Extract from File n8n node then converts the file into usable text data.


3. Primary AI Task (Summarization): The Summarise Agent (a LangChain LLM Chain powered by an OpenAI Chat Model) executes the core task, generating a summary (specifically, the top 5 highlights) from the transcript text.


4. Evaluation Check: An Is Evaluating? n8n node checks if the execution is a formal evaluation run. If not, the n8n workflow terminates the functional path (via Respond to User).


5. LLM Evaluation: If evaluation mode is active, the Evaluate Summarisation LangChain n8n node is invoked. This powerful n8n node sends both the original transcript and the generated summary to a specialized Google Gemini model (LLM). This evaluation LLM acts as an "expert judge," applying a comprehensive rubric (Groundedness, Conciseness, etc.) to assign a score (1-5).


6. Structured Scoring: The evaluation LLM's output is processed by the Output n8n node (Structured Output Parser) to guarantee a structured JSON result containing a numerical rating and a reason for the score.


7. Metric Logging: Finally, the Set Outputs and Set Metrics n8n nodes log the resulting score and score_reason back into the associated evaluation framework, completing the assessment cycle for this specific n8n workflow run.

1. Import: Copy the provided n8n workflow JSON code and import it into your n8n instance using the "New > Import from JSON" option.


2. Credentials Setup: This sophisticated n8n template requires several credentials:

1. Dataset Configuration: Configure the When fetching a dataset row n8n trigger to point to your specific evaluation spreadsheet and sheet name, ensuring the workflow knows where to pull test data from.


2. Activation: Ensure the n8n workflow is saved and activated.

When fetching a dataset row (Evaluation Trigger): The crucial n8n trigger for evaluation mode. It reads test cases row by row from a Google Sheet (e.g., "Summarization").
Download Transcript (Google Drive n8n node): Retrieves the required source material (transcript) from Google Drive using the input URL.
Summarise Agent (Chain LLM): The core AI engine. This n8n node uses the OpenAI model (e.g., gpt-4.1-mini) to generate the summary based on the prompt: "Summarise the top 5 highlights of this video using the provided transcript."
Is Evaluating? (Evaluation n8n node): A flow control mechanism determining if the n8n workflow is executing an evaluation job or a production run.
Evaluate Summarisation (Chain LLM): The scoring mechanism. This specialized n8n node uses the Google Gemini model and a detailed rubric to compare the generated summary against the source text for groundedness and quality.
Set Outputs & Set Metrics (Evaluation n8n node): These final evaluation n8n nodes capture the LLM judge's rating and reason, logging the resulting metrics (score and score_reason) back into the associated dataset via the evaluation framework provided by n8n.