Generate AI-Ready Files from Website Crawl Data - n8n Workflow

• SEO Specialists needing to structure website crawl data for AI systems.


• Data Engineers preparing website content for Large Language Model (LLM) training or RAG pipelines.


• n8n users seeking robust data parsing and file generation n8n templates.


• Developers utilizing n8n to automate complex data preparation tasks.

This n8n workflow streamlines the process of converting raw website crawl data into a structured llms.txt format, essential for optimizing content discovery by Large Language Models (LLMs) and other AI systems. Manually processing Screaming Frog exports can be time-consuming, but this specialized n8n template automates the entire preparation lifecycle.

By leveraging specific n8n node operations, the n8n workflow ensures high data quality by strictly filtering only indexable (Status Code 200, indexable) HTML pages. A key feature of this n8n workflow is its built-in multilingual compatibility, allowing it to interpret various international column names from Screaming Frog exports.

Furthermore, users have the option to enable an advanced AI filtering n8n node, utilizing OpenAI's Text Classifier to intelligently assess content quality based on title, description, and word count, ensuring only high-value pages are included in the final output. This powerful n8n workflow acts as a crucial bridge between SEO data collection and AI preparation.

The entire process is initiated via a user-friendly form n8n trigger, making this n8n workflow highly accessible:

1. Trigger and Upload: The Form - Screaming frog internal_html.csv upload n8n trigger starts the execution. Users submit the website name, a short description, and upload the Screaming Frog CSV export.


2. Data Extraction: The Extract data from Screaming Frog file n8n node reads the uploaded CSV, converting the row data into a stream of usable JSON items for the n8n workflow.


3. Data Standardization: The Set useful fields n8n node standardizes the data structure, mapping multilingual Screaming Frog column titles (e.g., 'Adresse', 'Title 1') to consistent fields like url, title, and indexability.


4. SEO Filtering: The Filter URLs n8n node applies strict quality control, retaining only items where the status code is 200, the indexability is 'Indexable', and the content type is 'text/html'.


5. Optional AI Filtering: An optional Text Classifier n8n node (powered by an OpenAI n8n node) can be activated to provide advanced, AI-driven filtering based on content relevance and quality indicators like word count and meta descriptions.


6. Row Formatting: The Set Field - llms.txt Row n8n node structures each remaining item into the required markdown format for llms.txt: - Title: Description.


7. Concatenation: The Summarize - Concatenate n8n node combines all individual rows into one massive text block, ensuring proper line breaks.


8. File Assembly: The Set Fields - llms.txt Content n8n node wraps the concatenated content with the website name and description gathered from the initial n8n trigger.


9. Output: Finally, the Generate llms.txt file n8n node converts this compiled text into a downloadable llms.txt file ready for use by AI systems. The final upload file anywhere n8n node is a placeholder, suggesting options to extend this n8n workflow to upload the file to cloud storage.

To use this n8n workflow, follow these steps:

1. Import: Copy the provided JSON code and paste it into your n8n instance using the 'Import from JSON' function.


2. Credentials: If you plan to activate the optional Text Classifier n8n node for AI filtering, you must configure your OpenAI API credentials within the corresponding n8n node settings.


3. Testing: Click 'Test Workflow' or run the active n8n workflow. The initial Form n8n trigger will prompt you to enter the required details and upload your Screaming Frog internal_html.csv file.


4. Execution: Once the workflow runs successfully, the final Generate llms.txt file n8n node will contain the output file, which you can download directly from the n8n UI, completing the automated generation process.

Form - Screaming frog internal_html.csv upload (n8n trigger): Acts as the entry point, collecting the website name, description, and the required Screaming Frog CSV export file.
Extract data from Screaming Frog file (n8n node): Responsible for parsing the binary CSV data into JSON objects, making the raw crawl data usable within the rest of the n8n workflow.
Set useful fields (n8n node): Performs crucial data mapping. It uses complex expressions to standardize field names like url, title, description, and statut, handling various language variations of Screaming Frog output.
Filter URLs (n8n node): Implements the core SEO logic by filtering items to ensure Status Code is 200, Indexability is Indexable, and Content Type is 'text/html'. This ensures only clean, crawlable URLs are processed by the n8n workflow.
Text Classifier (Optional n8n node): If activated, this LangChain-based n8n node connects to the OpenAI Chat Model to categorize pages based on content quality, enabling an 'intelligent' content filter for the llms.txt file.
Set Field - llms.txt Row (n8n node): Formats the remaining items into the precise markdown structure required for the llms.txt standard.
Summarize - Concatenate (n8n node): Aggregates all the individually formatted rows into a single text body, ensuring correct separation for the final file.
Generate llms.txt file (n8n node): The final action n8n node, converting the compiled text string into the downloadable file output.