Sitemap Crawler to Vector DB for RAG Knowledge Base - n8n Workflow

• AI Engineers building Retrieval Augmented Generation (RAG) systems.


• Developers needing to automatically populate a vector database from a website sitemap.


• Users looking for advanced n8n templates integrating scraping, OpenAI, and Supabase.


• Data architects requiring structured, cleaned website data for training large language models.

This comprehensive n8n workflow provides a robust solution for transforming raw website content into a ready-to-use knowledge base for AI agents. It tackles the common challenges of web scraping: handling sitemap structure, queuing URLs, cleaning noisy HTML, and ensuring data quality before vectorization. By leveraging a specialized scraping API (Crawl4AI) and integrating it with the OpenAI Embeddings n8n node, this n8n workflow automates the entire ETL (Extract, Transform, Load) process. The use of Supabase as both a job queue and a vector database ensures reliability and scalability for handling large scraping tasks, making this n8n template an essential component for production RAG systems built using n8n.

This powerful n8n workflow operates in three major phases: Sitemapping & Queuing, Scraping & Polling, and Vectorization.

1. Initial Setup: The workflow starts by executing two Postgres n8n node steps (intended for a one-time execution) to create the necessary scrapequeue and documents tables in Supabase, with the latter designed to hold high-dimensional vectors (1536).


2. Sitemap Processing: The n8n trigger, When clicking ‘Test workflow’, initiates the run. An HTTP Request n8n node fetches the site's sitemap, which the XML n8n node parses. The Split Out n8n node separates the sitemap into individual URLs.


3. URL Queuing: A Loop Over Items1 block iterates through all URLs. Supabase nodes check if the URL already exists in the scrapequeue. If the URL is new, it is inserted as 'pending' using the URL in a new row Supabase n8n node.


4. Scraping and Polling Loop: The main processing loop (Loop Over Items) retrieves a 'pending' URL and submits it to the Crawl4AI service via an HTTP Request n8n node, which returns a taskid.


5. Task Monitoring: The Wait n8n node pauses the execution, followed by the Crawl4AITask Status node, which checks the status. An If n8n node directs completed tasks to processing or sends pending tasks back to the wait loop, ensuring robust task management. Failed tasks are marked 'error' if they exceed 10 attempts (handled by the Taskid Counter Code n8n node).


6. Cleaning and Quality Control: The Remove redundant data from the scraping Code n8n node aggressively cleans the scraped content. Subsequent Code n8n nodes (Quality Filter Node, Content Type Detection, Better Metadata Extraction) validate data integrity, assign content types, and compile rich metadata.


7. Vectorization Pipeline: Cleaned documents are passed to the Character Text Splitter n8n node to break them into manageable chunks. The Embeddings OpenAI n8n node generates the vector embeddings.


8. Final Storage: The Supabase Vector Store n8n node then inserts the resulting text chunks, metadata, and embeddings into the documents vector database table. The status in the scrapequeue is updated to 'completed', finalizing the iteration of this comprehensive n8n workflow.

To set up this advanced n8n template, follow these steps:

1. Import: Copy the provided n8n workflow JSON and import it into your n8n instance.


2. Supabase Credentials: Configure the required Supabase API credentials. This n8n workflow requires credentials for the SupabaseN8N AI Agent Assistantmarinextai service, connecting to your Supabase project.


3. Postgres/Vector Configuration: Ensure your Supabase instance has the necessary Postgres extensions enabled (like vector). You must also configure the Postgres credential (PostgresN8N AI Agent Assistantmarinextai). The two initial Postgres n8n nodes must be run once manually to create the scrapequeue and documents tables.


4. OpenAI Credentials: Set up the API key for the Embeddings OpenAI n8n node, specifically the OpenAiMariela.b.d. credential, which uses the text-embedding-ada-002 model.


5. Crawl4AI Configuration: Configure the Generic HTTP Header Authentication credential (Crawl4ai_marinextai) needed by the Crawl4AI HTTP Request n8n node steps.


6. Testing: Execute the initial When clicking ‘Test workflow’ n8n trigger to start fetching the sitemap and populating the queue.

Manual Trigger (When clicking ‘Test workflow’): The initiating n8n trigger for this manual workflow run.
HTTP Request (Sitemap): Fetches the target website’s sitemap.xml file.
XML n8n node: Parses the received XML content from the sitemap into a usable JSON structure.
Supabase n8n node (Check/Insert URL): Used extensively for URL queue management, ensuring that URLs from the sitemap are checked for existence and inserted into the scrapequeue only if they are new or pending.
HTTP Request (Crawl4ai Web Page Scrape): Submits the URL to the external scraping service and retrieves a unique taskid.
Code n8n node (Remove redundant data): A critical step in the n8n workflow that uses custom JavaScript to clean HTML and Markdown noise from the scraped content, optimizing it for RAG.
Code n8n node (Quality Filter/Metadata): Filters out low-quality or non-meaningful pages and extracts comprehensive metadata (title, domain, language).
Character Text Splitter n8n node (LangChain): Splits the large, cleaned text into smaller chunks of 5000 characters suitable for embedding.
Embeddings OpenAI n8n node: Generates 1536-dimensional vector embeddings for each text chunk using OpenAI's API.
Supabase Vector Store n8n node (documents): Inserts the final structured data, including the text chunks, metadata, and the newly generated vector embeddings, into the Supabase vector database. This specialized n8n node is vital for RAG setup.

• Wait n8n node: Implements a time delay of 30 seconds necessary for polling the status of the asynchronous web scraping task.