AI-Powered Structured News Scraping and Database Logging - n8n Workflow

Data journalists needing structured daily news feeds.
Market researchers tracking competitive intelligence from specific forums.
AI enthusiasts looking for practical n8n templates for large language model (LLM) integration.
Users who require a scheduled n8n workflow for continuous data ingestion.

This sophisticated n8n workflow provides a complete solution for high-volume, structured data extraction. The inherent challenge in web scraping is dealing with unstructured or poorly formatted text output. This powerful n8n solution overcomes this by integrating a dedicated scraping tool (Decodo) with a cutting-edge LLM (Google Gemini).

The flow is designed to run periodically via an n8n trigger, systematically cycling through multiple target URLs. For each scraped page, the raw content is passed to the Gemini AI n8n node, which executes a precise extraction prompt, converting the messy text into a clean, schema-enforced JSON array. Finally, this structured data is automatically appended or updated in a Google Sheet, turning raw web data into an organized, queryable database. This complete n8n workflow saves countless hours of manual data cleanup and is one of our most popular n8n templates for data management.

The process begins with the Schedule Trigger, acting as the main n8n trigger, initiating the data collection loop.

1. Configuration: The Workflow Config n8n node defines the target forum URLs and geolocation parameters.


2. Iteration: The Split Forums and Iterate Forums nodes ensure that the subsequent scraping steps run individually for every URL configured in the initial setup.


3. Scraping: The Scrape Forum Data Decodo n8n node visits the defined URL and extracts the raw text content.


4. AI Structuring: The raw text is passed to the LangChain Extract Structured News Data n8n node, which uses the Google Gemini Model to transform the unstructured text into a strict JSON array based on a predefined schema (title, url, source, points, etc.).


5. Logging & Delay: A logging step (Log Scrape Results Google Sheets n8n node) records the success and count of items extracted for the specific forum URL. A Wait Between Scrapes n8n node then introduces a brief 1-minute pause before the next iteration begins, managing API usage efficiently.


6. Item Processing: The structured JSON output is broken down into individual news items using the Split News Items n8n node.


7. Unique Key: The Generate Unique Key n8n node creates a unique ID for each news item, crucial for managing data integrity in the database.


8. Storage: The Update Google Sheet (News) n8n node performs an appendOrUpdate operation in Google Sheets, ensuring that news items are updated if the unique key already exists, or appended if new. This complete n8n workflow ensures reliable data storage.

To set up this advanced n8n workflow, follow these steps:

1. Import the n8n Workflow: Copy the provided JSON code and paste it directly into your n8n instance using the "Import Workflow" function, available when creating a new n8n workflow.


2. Credentials Setup: You must configure credentials for three services:

1. Google Sheet Preparation: Ensure your target Google Sheet tab (News) has the required header columns: key, title, url, source, points, comments, author, postedat, and lastupdated. Also, ensure the Logs tab is set up correctly.


2. Activate: Set your preferred interval in the Schedule Trigger n8n trigger node and activate the n8n workflow.

This n8n workflow relies on several specialized n8n nodes for scraping, AI processing, and storage:

Schedule Trigger (n8n trigger):
Function: Starts the entire n8n workflow automatically at defined intervals.
Key Configuration: Set to run periodically (e.g., daily or hourly).
Workflow Config (Set n8n node):
Function: Centralizes configuration variables, including target URLs (forums) and the Google Sheet ID (sheet_id), making this n8n template easily adaptable.
Scrape Forum Data (Decodo n8n node):
Function: Executes the external scraping service to fetch the raw text content from the configured forum URL.
Key Configuration: Uses expressions to pull url and geo from the input data.
Extract Structured News Data (LangChain LLM Chain n8n node):
Function: The primary intelligence step. It sends the raw scraped text to the AI model (Gemini) with a highly specific system prompt to extract structured data strictly in JSON format (title, url, author, etc.).
Key Configuration: Connected to the Google Gemini Model and uses a detailed system message to define the JSON output schema.
Log Scrape Results (Google Sheets n8n node):
Function: Logs the success and item count for each individual forum scrape into a Logs tab for monitoring.
Key Configuration: Operation set to append.
Split News Items (Split Out n8n node):
Function: Converts the single JSON array output by the AI into multiple individual n8n items, allowing downstream processing of each news post independently.
Generate Unique Key (Crypto n8n node):
Function: Creates a deterministic, unique identifier by combining the item's URL and author. This is essential for the appendOrUpdate logic.
Update Google Sheet (News) (Google Sheets n8n node):
Function: The final step of this n8n workflow. It writes the structured data to the Google Sheet, using the key to identify and update existing records or append new ones.
* Key Configuration: Operation is set to appendOrUpdate, matching on the key column.