Auto-RAG Pipeline for Document Processing using Pinecone and Google Drive - n8n Workflow

• AI Engineers and Developers building RAG (Retrieval-Augmented Generation) applications.


• Organizations dealing with large volumes of scanned or OCR-processed documents.


• Users needing a robust n8n workflow to handle complex text processing, normalization (especially Arabic script), and vector database ingestion.


• Technical teams looking for advanced n8n templates integrating Google Drive, OpenAI, and Pinecone.

Manually processing large batches of OCR output files (like those generated by Google Vision or Document AI) and preparing them for a vector database is tedious and prone to error. This sophisticated n8n workflow solves this by providing a zero-touch pipeline. It automatically detects new OCR JSON files in a designated Google Drive folder. Crucially, the n8n template uses custom JavaScript logic to extract essential lesson metadata directly from the filename, clean and normalize the complex Arabic text, and chunk the content efficiently. By integrating the OpenAI n8n node for high-quality embeddings and the Pinecone n8n node for storage, this n8n workflow creates a reliable and scalable foundation for semantic search and AI knowledge retrieval, concluding by archiving the source file.

This automation begins with the Watch Drive Folder (new files) n8n trigger. When a new file appears in the input Google Drive folder, the n8n workflow immediately starts executing.

1. Trigger & Metadata Extraction: The Google Drive n8n trigger identifies the new file. The subsequent Code n8n node, 'Filename → Lesson Metadata', analyzes the filename using regex to extract structured metadata (grade, lesson number, subject, title, etc.), preparing key fields like the Pinecone namespace.


2. File Download & Parsing: The file is downloaded via the Google Drive n8n node. The 'Vision JSON → Clean Text Chunks' Code n8n node performs the core processing. It parses the OCR JSON structure (handling Vision or Document AI formats), extracts the raw text, applies custom Arabic text normalization functions (removing diacritics, standardizing characters), removes common noise (headers, watermarks), and splits the text into optimal, overlapping chunks (900 character max, 150 overlap).


3. Embedding Generation: Each resulting text chunk is passed to the 'Generate Embeddings (OpenAI)' n8n node, which uses the configured OpenAI service to convert the text into dense vector representations.


4. Vector Store Insertion: The 'Insert into Pinecone Vector Store' n8n node takes the text, its associated metadata (parsed in step 1), and the new vector embeddings, inserting them into the designated Pinecone index and namespace. This is where the RAG knowledge base is built.


5. Cleanup: Finally, the 'Move File to Archive' Google Drive n8n node moves the original JSON file to a specified archive folder, ensuring the main ingestion folder remains clear for the next execution of the n8n trigger.

To deploy this powerful n8n workflow, follow these steps:

1. Import: Copy the provided JSON definition and import it as a new n8n workflow.


2. Credentials: Set up or link the required credentials:

1. Activate: Save the n8n workflow and set the status to 'Active'.

Watch Drive Folder (new files) (n8n trigger):
Function: Polls a specific Google Drive folder for newly created files.
Key Configuration: Event: fileCreated, Trigger On: specificFolder, set to poll every minute.
Filename → Lesson Metadata (Code n8n node):
Function: Custom JavaScript logic to parse complex, structured filenames (e.g., arabicg12arlesson01...) into clean, usable metadata fields (e.g., titlear, bookid, namespace).
Key Configuration: Contains extensive regex and string manipulation for robust data extraction.
Vision JSON → Clean Text Chunks (Code n8n node):
Function: The central processing n8n node. Reads the base64-encoded OCR JSON file, extracts the full text annotation, normalizes Arabic characters, cleans up document noise, and implements a semantic chunking algorithm.
Key Configuration: Custom functions for normalizeAr and makeChunks (max size 900, overlap 150).
Generate Embeddings (OpenAI) (LangChain n8n node):
Function: Generates high-dimensional vector embeddings for each text chunk using the OpenAI API.
Key Configuration: Uses the credentials defined for OpenAI.
Insert into Pinecone Vector Store (LangChain n8n node):
Function: Uploads the text, embeddings, and detailed metadata to the specified Pinecone index and dynamically determined namespace.
Key Configuration: Mode: insert, dynamically sets pineconeNamespace based on metadata, Embedding Batch Size: 64 for efficient ingestion.
Move File to Archive (Google Drive n8n node):
Function: Moves the successfully processed file to a designated archive location.
Key Configuration: Operation: move, uses the ID from the initial n8n trigger output.