Dynamic Knowledge Base Generation from Web Content - n8n Workflow

• AI/ML Engineers needing clean, structured data for Retrieval-Augmented Generation (RAG) systems.


• Content managers automating the conversion of websites into internal training documents.


• Developers looking for reusable n8n templates for complex multi-step data processing.


• Anyone who needs to consistently convert dynamic web content into LLM-friendly text files.

Building a reliable knowledge base for Large Language Models requires meticulously clean and formatted source material. Standard web scraping often results in noisy HTML. This sophisticated n8n workflow solves this problem by integrating specialized services like Firecrawl (for deep URL mapping) and Parsera (for clean content extraction) with AI processing via the OpenAI n8n node.

This n8n template is triggered by a form submission, allowing users to easily input target URLs. The n8n node structure then intelligently handles both single URL extractions and recursive batch processing. By utilizing the openAi n8n node to format the content, we ensure the final output is optimized for LLM consumption before being securely uploaded to Google Drive. This end-to-end n8n automation significantly accelerates the data preparation pipeline for RAG and custom GPTs.

This comprehensive n8n workflow begins with the Trigger — Form (Create LLM KB) n8n trigger, which receives the target URL(s) from a user input form.

1. Decision Routing: The Decision n8n node evaluates whether the input requires batch processing (e.g., if recursive crawling is requested or multiple URLs are provided).


2. URL Mapping (Batch Path): For complex inputs, the Firecrawl — Map URLs HTTP Request n8n node recursively finds associated sub-pages. These URLs are then split and batched using the Split URLs and Batch URL Processor n8n nodes.


3. Extraction & Cleaning: Whether processing a single URL or a batch, the respective Extract Markdown (Parsera) HTTP Request n8n node cleans the HTML and converts the content into structured Markdown.


4. AI Formatting: The dedicated LLMs.txt Generator (OpenAI n8n node) takes the clean Markdown. Using advanced LLM capabilities (e.g., GPT-4), it standardizes the text, summarizing or optimizing it according to best practices for LLM training data. This step is crucial for the quality of the final knowledge base.


5. File Preparation: The processed output is prepared using the File Fields (Set n8n node), and then the Convert to TXT n8n node packages the data into a standard .txt file format.


6. Storage: Finally, the Google Drive — Upload to folder n8n node saves the newly created, LLM-ready text file(s) into the designated cloud storage location, completing the automated n8n workflow.

To deploy this powerful n8n workflow, follow these steps:

1. Import the n8n template: Copy the provided JSON data and import it directly into your n8n instance.


2. Setup Credentials: You must configure credentials for the following services within your n8n environment:

1. Configure the n8n Trigger: Activate the Trigger — Form n8n trigger and note its webhook URL. This is the link users will use to start the process.


2. Customize Nodes: Review the Decision n8n node logic and ensure the pathing and batch sizes in the Batch URL Processor n8n node meet your specific requirements. Customize the prompt in the LLMs.txt Generator (OpenAI n8n node) to tailor the output formatting for your specific LLM or RAG application.

Trigger — Form (Create LLM KB): The primary n8n trigger. Starts the n8n workflow upon submission of a web form, gathering input URLs.
Decision — Generate For (Switch n8n node): Controls flow based on whether a single URL or a batch/recursive crawl is required, routing the data to the appropriate processing path within the n8n template.
Firecrawl — Map URLs (HTTP Request n8n node): In the batch path, this external API call maps out recursive links from the starting URL to ensure a comprehensive capture of the site's structure.
Extract Markdown (Parsera - HTTP Request n8n node): Cleans and extracts the primary textual content from the raw webpage into structured Markdown, greatly improving the quality of the input for the subsequent LLM n8n node.
LLMs.txt Generator (OpenAI n8n node): Utilizes the powerful OpenAI integration in this n8n workflow. It takes the cleaned text and applies AI logic (summarization, structural refinement) to generate an ideal knowledge base file.
Convert to TXT (Convert File n8n node): An essential n8n node for data preparation, ensuring the output from the AI process is correctly encoded and saved as a plain text file.

• Google Drive — Upload to folder (Google Drive n8n node): The final action in this n8n automation, responsible for storing the generated, high-quality knowledge base file(s) in a designated cloud folder for easy access by RAG systems.