AI Agent Prompt A/B Split Testing with Supabase and OpenAI - n8n Workflow

AI Developers and Researchers needing to benchmark LLM prompt performance.
Data Scientists implementing controlled experiments in production environments.
MLOps Engineers seeking robust testing methodologies within their automation pipelines.
Users looking for advanced examples of conditional logic within an n8n workflow.

• Anyone needing persistent session management in an n8n automation using Supabase.

Optimizing the performance of Large Language Models (LLMs) often requires rigorous testing of the system prompts they use. This n8n workflow provides a structured framework for conducting A/B split tests on different prompts. By integrating Supabase, the workflow ensures that once a user's session ID is assigned to a prompt variant (either 'baseline' or 'alternative'), they consistently receive that prompt throughout their conversation, ensuring clean test data.

The value of this automation lies in its ability to enable data-driven decisions regarding prompt engineering. Instead of guessing which prompt performs better, you can deploy this n8n template, measure metrics (which you can add as a subsequent step), and confirm efficacy. The automation starts with a dedicated n8n trigger for chat messages, making it seamless for integration into chat applications.

This n8n workflow begins execution when a new chat message is received by the When chat message received n8n trigger.

1. Define Prompts: The Define Path Values n8n node first sets up the static text values for both the 'baseline' and 'alternative' prompts that will be tested.


2. Session Check: The workflow then uses a Supabase n8n node (Check If Session Exists) to query the splittestsessions table using the unique sessionId provided by the initial n8n trigger. This checks if the user has been assigned a path previously.


3. Conditional Split: The If Session Does Exist n8n node evaluates the Supabase output.

1. Get Correct Prompt: The Get Correct Prompt Set n8n node utilizes the stored or newly generated showalternative boolean to dynamically select the correct prompt (either baselinevalue or alternativevalue). This final prompt is used for the AI Agent.


2. Agent Execution: The AI Agent n8n node takes the user input and the selected system prompt. It uses the OpenAI Chat Model (specifically configured here for gpt-4o-mini) and Postgres Chat Memory (backed by Supabase) to generate a contextually relevant response, completing the cycle of this sophisticated n8n workflow.

To successfully deploy this powerful n8n workflow, follow these setup steps:

1. Import: Import this n8n workflow JSON into your n8n instance.


2. Supabase Setup: You must have a Supabase database configured. Create a table named splittestsessions containing at least two columns:

1. Credential Configuration:

1. Prompt Modification: Edit the Define Path Values n8n node. Update the baselinevalue and alternativevalue fields with the actual system messages you wish to A/B test.


2. Activation: Activate the n8n workflow and test using n8n’s built-in chat interface to see both paths in action.

This n8n workflow leverages specialized nodes for AI interaction, database management, and flow control:

When chat message received (n8n trigger):
Function: Serves as the starting point (the n8n trigger) for the automation, capturing the user's chatInput and unique sessionId.
Key Configuration: Langchain Chat Trigger type.
Define Path Values (n8n node):
Function: A Set n8n node used to store the static text variables for the A and B prompt variants (baselinevalue and alternativevalue).
Key Configuration: Defines two string variables for the test prompts.
Check If Session Exists (n8n node):
Function: Queries the Supabase splittestsessions table using the sessionId to determine if a prompt path has already been assigned.
Key Configuration: Operation set to get, filtering by sessionid.
If Session Does Exist (n8n node):
Function: Directs the flow based on the existence of a session record, ensuring new users are randomly assigned a path.
Key Configuration: Checks if an item ID exists in the incoming data from Supabase.
Assign Path To Session (n8n node):
Function: For new sessions, this Supabase n8n node inserts a record, randomly setting the showalternative field to true or false using JavaScript (Math.random() < 0.5).
Key Configuration: Operation set to insert, table splittestsessions.
Get Correct Prompt (n8n node):
Function: Determines the final system prompt by conditionally selecting the baselinevalue or alternativevalue based on the showalternative boolean.
Key Configuration: Uses an expression like {{ $json.showalternative ? alternativevalue : baselinevalue }}.
AI Agent (n8n node):
Function: The central execution node that processes the chat input using the selected prompt variant, connected to the LLM and memory components.
Key Configuration: System message dynamically set to the determined prompt variable.
OpenAI Chat Model (n8n node):
Function: Provides the computational power for the AI Agent, configured here to use a cost-effective model like gpt-4o-mini.
Postgres Chat Memory (n8n node):
* Function: Manages conversational state by storing and retrieving chat history in a PostgreSQL table, keyed by the sessionId.