AI-Powered Multi-Source Article URL Discovery Engine - n8n Workflow

SEO professionals needing to monitor content output from competitors or niche publishers.
Data analysts requiring high-quality, normalized content links for large-scale analysis.
Content aggregators who need reliable, structured input data.
Anyone looking for advanced web scraping techniques leveraging the power of an n8n node combined with AI intelligence.

This sophisticated n8n workflow template solves the challenge of discovering specific, high-value article links hidden within complex websites that typical scraping tools might miss or misinterpret. By integrating the n8n HTTP Request node with an advanced OpenAI LangChain n8n node, this solution ensures robust extraction. The primary n8n trigger can be set daily, allowing continuous monitoring of seed websites. The core value lies in the AI's ability to differentiate valid articles from noise (like navigation, category pages, or PDFs), providing clean, normalized data stored conveniently in Google Sheets. If you are searching for robust n8n templates for data enrichment and web scraping, this AI-driven n8n workflow provides unparalleled precision.

1. Trigger and Input: The n8n workflow initiates either manually or via the daily scheduled n8n trigger. It uses the Google Sheets n8n node to retrieve the initial list of seed URLs designated for crawling.


2. Rate-Limited Loop: The Loop Over URLs n8n node processes each URL sequentially, using a 3-second Wait n8n node to prevent aggressive scraping and potential rate limiting from target sites.


3. Fetching and Conversion: The Fetch Webpage HTML n8n node performs the scrape, spoofing a browser user-agent for successful retrieval. The raw HTML content is then passed to the HTML to Markdown n8n node for optimal clean-up before AI processing.


4. AI Extraction: The central AI URL Extractor n8n node (powered by the OpenAI Chat Model) analyzes the page content against strict, predefined rules. This dedicated n8n node identifies only legitimate article links, converts relative paths to absolute URLs, removes tracking parameters, and assigns a precise source category.


5. Normalization and Deduplication: A specialized Code n8n node handles the post-processing of the AI output. It strips markdown code fences, parses the resulting JSON array, performs final URL normalization (e.g., removing tracking queries), and ensures only unique links proceed to the final stage of the n8n workflow.


6. Output: Finally, the results are stored via the Google Sheets n8n node, appending the discovered URLs and their assigned sources to the designated output sheet, completing the n8n workflow execution.

1. Import the n8n workflow: Copy the provided JSON data and paste it into your n8n instance using the "Import from JSON" function.


2. Credentials Setup: You must configure two sets of credentials:

1. Google Sheets Preparation: In your target spreadsheet, ensure you have an input sheet (e.g., "Seed URLs") with a column titled URL. Create an output sheet (e.g., "Discovered URLs") that will receive the final, structured data from the n8n template.


2. n8n Node Configuration: Update the Read Seed URLs and Save Discovered URLs n8n nodes to accurately reference your specific Google Sheet Document ID and the corresponding sheet names.


3. Activation: Activate the n8n workflow and run it using the Manual Trigger or rely on the scheduled n8n trigger to begin the process.

Daily Schedule (6 AM) / Manual Trigger: The starting n8n trigger points. Provides flexibility to run the n8n workflow on demand or at a fixed time.
Read Seed URLs (Google Sheets n8n node): Function: Reads the initial list of URLs from the input Google Sheet. Key Configuration: Targets the URL column of the designated input sheet.
Rate Limit (Wait n8n node): Function: Implements a 3-second mandatory delay between processing each URL to prevent bot detection and rate limiting.
Fetch Webpage HTML (HTTP Request n8n node): Function: Downloads the HTML content of the target page. Key Configuration: Uses custom User-Agent headers to ensure reliable scraping by mimicking a standard browser; set to continue on failure.
HTML to Markdown (Markdown n8n node): Function: Converts the raw HTML into a cleaner Markdown format, optimizing the input for the downstream AI n8n node processing.
AI URL Extractor (LangChain Agent n8n node): Function: The core AI n8n node that uses the system prompt to intelligently filter, identify, normalize, and categorize legitimate article URLs from the provided text using the OpenAI n8n node. Key Configuration: Requires strict adherence to JSON output and detailed exclusion rules (e.g., skipping category pages).
URL Parser & Normalizer (Code n8n node): Function: Cleans and validates the AI's JSON output, removes tracking queries, and performs the final list deduplication before saving the results. This custom n8n node ensures data integrity.
Save Discovered URLs (Google Sheets n8n node): Function: Stores the final, unique list of discovered article URLs and their source categories back into the specified output sheet. Key Configuration: Uses the appendOrUpdate operation.