Domain-Specific Recursive Web Content Scraper - n8n Workflow
Use this powerful n8n workflow template for deep, domain-limited web crawling. Extracts text, manages recursion with n8n static data, and chunks output.
Workflow Preview
Ready to automate?
Download this n8n workflow template and start using it instantly.
Who is this best for?
- Developers building Custom Web Scraping Tools.
- Content Managers needing to archive site data into a structured format.
- AI/RAG engineers preparing large, domain-specific text corpora.
- n8n users seeking advanced examples of recursive logic using the n8n node structure.
Overview
Achieving controlled, recursive crawling within an automation platform requires robust state management. This sophisticated n8n workflow solves this challenge by leveraging n8n's static data features to maintain the crawl state, including tracking visited URLs, managing the pending queue, and controlling the crawl depth. This specific n8n workflow template is designed to strictly adhere to the initial domain, preventing external link following, and automatically filters out non-HTML files like PDFs and DOCXs. The entire process, driven by the core n8n node logic, ensures efficiency and control, culminating in a structured, batched output that is ideal for downstream processing like AI analysis or database ingestion.
How it Works
This comprehensive n8n workflow utilizes an advanced looping mechanism combined with static data tracking to execute the crawl:
- Trigger and Initialization: The process begins with the
Webhookn8n trigger, which accepts the starting URL via a POST request. The subsequentInit Crawl Paramsn8n node defines the initial settings, includingmaxDepth(set to 3). TheInit GlobalsCode n8n node initializes the crucial static data variables (visitedlist,queuedlinks dictionary,pendingcount, andpagesarray) needed for the recursive nature of the n8n workflow. - Starting the Loop: The
Seed Root Crawl ItemMerge n8n node combines the initial parameters, sending the root URL into the fetching pipeline. - Fetch and Extraction: The
Fetch HTML PageHTTP Request n8n node fetches the raw HTML content. TheExtract Body & LinksHTML n8n node then cleans the content, extracts the body text, and identifies all anchor links (tags). - Deduplication and Queueing: The core intelligence resides in the
Queue & Dedup LinksCode n8n node. This custom n8n node normalizes all extracted links, checks against thevisitedlist, enforces the same-domain rule, and adds eligible, unvisited links to the globalqueuedlist. It carefully manages the static datapendingcounter to track pages remaining to be processed. - Depth Control and Recursion: The
IF Crawl Depth OK?n8n node checks if the link's depth is less than or equal tomaxDepth. If the condition is met, the link is sent through theRequeue Link Itemand then theLoop Links (Batches)SplitInBatches n8n node, which iterates the queue back into the start of the fetching pipeline, completing the recursive n8n workflow loop. - Data Storage and Termination: Successfully processed pages are captured by the
Store Page Datan8n node. TheCollect Pages & Emit When DoneCode n8n node checks thependingcounter. Oncependingreaches zero, this n8n node emits the complete, aggregated content string. - Final Output: The
Combine & ChunkCode n8n node packages the collected data, splitting it into manageable batches (chunks) based on character limits, ensuring the final output is delivered reliably via theRespond to Webhookn8n node.
Installation Guide
To deploy this powerful web scraping n8n workflow template, follow these steps:
- Import: Copy the provided JSON data and paste it directly into your n8n instance using the 'New' -> 'Import from JSON' option.
- Webhook Configuration: The
Webhookn8n trigger is automatically configured with a unique ID. Set the n8n workflow to 'Active'. You will use the displayed testing or production URL to trigger the crawl. - Credentials: The
Fetch HTML Pagen8n node uses a standard HTTP Request and typically requires no specialized credentials unless the target site is protected. - Trigger the n8n Workflow: Send a POST request to the webhook URL with a JSON payload containing the start URL, e.g.,
{"url": "https://www.example.com"}. - Adjusting Limits: To change the crawl depth, modify the
maxDepthvalue within theInit Crawl ParamsSet n8n node.
Node Details
Webhook (n8n trigger): The entry point for the n8n workflow. Configured to accept a POST request, initiating the crawl based on the provided starting URL.
Init Crawl Params (Set n8n node): Sets initial parameters like the root url, domain, and crucially, the maxDepth (default: 3).
Init Globals (Code n8n node): Initializes the static data storage for the entire n8n workflow, defining global variables for state tracking (pending, visited, queued, pages).
Fetch HTML Page (HTTP Request n8n node): Performs the actual HTTP GET request to retrieve the page. Configured with a 5-second timeout and continues execution even upon error (onError: continueRegularOutput).
Extract Body & Links (HTML n8n node): Extracts the clean body content and an array of links (href attributes) from the fetched HTML.
Queue & Dedup Links (Code n8n node): The central logic n8n node. It manages URL normalization, deduping against visited and queued lists, ensuring same-site restriction, and updating the static pending count.
IF Crawl Depth OK? (IF n8n node): Checks if the item is a 'link' and if the current depth is within the maxDepth set in the initial parameters. This directs the flow for recursion or storage.
Loop Links (Batches) (SplitInBatches n8n node): Essential for recursion. It processes links one by one (batchSize: 1), feeding them back to the start of the loop (Seed Root Crawl Item Merge n8n node).
Collect Pages & Emit When Done (Code n8n node): Appends the processed page data to the global pages array. This n8n node checks the static pending counter for termination. If zero, it outputs the combined content, finalizing the n8n workflow's core task.
Respond to Webhook (RespondToWebhook n8n node): Sends the final, aggregated, and chunked results back to the system that initiated the n8n trigger.
Related n8n Workflows
Domain and Email Contact Enrichment using Apollo API
Use this powerful n8n workflow to automate B2B lead enrichment. Extract emails and contact data from domains stored in Google Sheets using the Apollo API.
AI-Powered Lead Generation and Hyper-Personalized Outreach Automation
Deploy an advanced n8n workflow for automated lead generation, web content scraping via Jina AI, AI-powered personalization using OpenAI, and automated Gmail outreach.
Domain Availability Sniper with Porkbun & Multi-Channel Alerts
Use this powerful n8n workflow to automatically monitor domain name availability using the Porkbun API. Get instant Gmail and Discord alerts when a domain drops.
Facebook Video to Google Drive Converter with Sheets Logging
Automate Facebook video downloading and storage using this robust n8n workflow. Upload MP4 files to Google Drive and log every transaction in Google Sheets with built-in error handling.
Telegram Bot for Facebook & Instagram Video/Reel Downloader
Use this powerful n8n workflow to create a Telegram bot that automatically downloads Facebook and Instagram videos or Reels when a user pastes a link. Explore n8n templates for social media automation.
Salesforce Architect with 10+ years of experience in CRM, integrations, and automation. Skilled in Apex, LWC, REST APIs, and full-stack dev (JavaScript, .NET). I build secure, scalable workflows in n8n—connecting Salesforce, Stripe, and more. Passionate about lead scoring, data sync, and secure field masking. Certified Application Architect with deep expertise in platform, integration, and data architecture.
Featured*
