Recursive Website Content Collector using Airtop and Google Docs - n8n Workflow

Content Researchers: Individuals needing to build comprehensive knowledge bases or RAG systems by extracting large amounts of targeted content.
SEO Specialists: Users tracking website architecture and content density across many linked pages.
Developers & Integrators: Professionals looking for robust n8n templates for recursive data acquisition.
Anyone needing recursive web scraping: Users who require a reliable n8n workflow solution for multi-level site crawls.

This comprehensive n8n workflow is designed to automate the challenging task of recursive web scraping. Instead of manually clicking through a site, this n8n template uses the Airtop n8n node to intelligently crawl links up to a user-defined depth.

It provides a centralized system for tracking scraped URLs (using Google Sheets) and storing the resulting content (in Google Docs), making it one of the most practical n8n templates for serious data collection.

By leveraging the On form submission or Execute Workflow as the n8n trigger, users can specify the starting URL, the necessary link inclusion string, and the desired scraping depth, creating a truly dynamic and repeatable n8n workflow.

This powerful n8n workflow executes in several distinct phases, managing state using Google Sheets to prevent redundant scraping and track depth:

1. Trigger and Initialization: The n8n trigger (On form submission or When Executed by Another Workflow) accepts the 'Seed url', 'Links must contain' filter, and 'Depth'. The workflow then uses the Unify params n8n node to standardize the input.


2. Setup Tracking: A new Google Sheet (Create Spreadsheet) is generated to track all found URLs and their scraping status. The initial seed URL is immediately appended (Load info to spreadsheet).


3. Initial Scrape: The primary URL is scraped using the Scrape webpage Airtop n8n node. A new Google Doc is created, and the initial scraped content is written to it.


4. Recursive Check: The Should scrape more? n8n node (an If node) checks if the current iteration count ($runIndex) is less than the defined depth minus one. If true, the recursive loop continues.


5. Link Discovery: The workflow reads all existing links from the Google Sheet (Read scraped webpages). The Airtop n8n node (Retrieve links to scrape) is used to intelligently query the recently scraped page for new internal links.


6. Filtering and Deduping: A specialized Code n8n node (Filter links to insert to Sheets) processes the extracted links, ensuring they contain the required filter string and removing any duplicates already listed in the Google Sheet.


7. Updating Sheets: New, unique links are appended to the Google Sheet (Insert new links).


8. Deep Scraping and Document Update: The workflow then iterates through the newly added links, using the Airtop n8n node (Scrape webpage1) to extract content and appending this content to the centralized Google Doc (Update with new scraped content).


9. Flagging Status: Finally, the scraped URL is flagged in the Google Sheet (Insert flag) to mark the current iteration, concluding one loop of the recursive process and preparing for the next depth level, demonstrating the efficiency of this n8n workflow.

To deploy this efficient n8n workflow, follow these steps:

1. Import: Copy the provided n8n workflow JSON and paste it into your n8n instance.


2. Airtop Credentials: Set up credentials for the Airtop n8n node. You will need a valid Airtop API Key.


3. Google Sheets/Docs Credentials: Configure your Google Sheets and Google Docs OAuth2 API credentials. This often involves creating a project in the Google Cloud Console and enabling the necessary APIs (Drive, Sheets, and Docs) for your n8n setup.


4. n8n Trigger Activation: Enable the workflow. You can trigger it either via the integrated form or by calling the When Executed by Another Workflow n8n trigger from another n8n workflow.

On form submission / When Executed by Another Workflow (n8n Trigger): Serves as the starting n8n trigger. Configured to accept three essential parameters: Seed url, Links must contain (string filter), and Depth (number of recursive levels).
Create Spreadsheet (Google Sheets n8n node): Initializes a new Google Sheet named dynamically using the seed URL, serving as the central URL registry for this n8n workflow.
Load info to spreadsheet (Google Sheets n8n node): Appends the initial Seed URL to the newly created tracking sheet.
Scrape webpage / Scrape webpage1 (Airtop n8n node): Performs the heavy lifting of web extraction. Uses the 'extraction' resource and 'scrape' operation to pull content from the target URL.
Create Google Docs / Write scraped content (Google Docs n8n node): Manages the central document where all scraped text is aggregated. The initial setup creates the document, and subsequent steps use the 'update' operation to append more content.
Should scrape more? (If n8n node): The core logic for recursion. It compares the current run index ($runIndex) against the target depth to control the looping process, ensuring the n8n workflow stops at the specified level.
Retrieve links to scrape (Airtop n8n node): A smart n8n node using the 'query' operation and a JSON schema output to extract structured internal links from the scraped page content.
Filter links to insert to Sheets (Code n8n node): Custom JavaScript logic that handles filtering (based on the input string) and advanced deduplication against all links already present in the Google Sheet, optimizing the recursive crawl for this n8n workflow.