Structured Data Mining and Topic Modeling with Bright Data and Google Gemini - n8n Workflow

Data Scientists and Analysts needing structured data from unstructured web sources.
Developers looking for powerful n8n templates for advanced web scraping.
Users requiring automated market research and trend monitoring.
Anyone interested in integrating Bright Data and Google Gemini into a robust n8n workflow.

This comprehensive n8n workflow demonstrates high-level integration between premium web scraping services (Bright Data) and advanced Large Language Models (Google Gemini). The challenge of transforming raw, unstructured HTML or Markdown into actionable, clean, structured JSON data is solved efficiently. This specific n8n workflow fetches content from a target URL, cleans the output using a specialized n8n node for markdown conversion, and then bifurcates the processing to perform dual, structured AI analyses: topic extraction and emerging trend clustering. This powerful setup is a perfect example of modern data mining using n8n. Utilizing this n8n template accelerates the process of extracting complex insights directly from the web, providing ready-to-use structured data files.

1. Trigger and Setup: The process begins with a manual n8n trigger. The subsequent "Set URL and Bright Data Zone" n8n node defines the target URL (e.g., a news site) and the specific Bright Data Web Unlocker zone to be used in the n8n workflow.


2. Web Request: The "Perform Bright Data Web Request" n8n node executes an HTTP POST request to the Bright Data API, using HTTP Header Authentication to retrieve the webpage content, specifically requesting the output be formatted as Markdown.


3. Data Cleaning (LLM Chain): The raw markdown data is passed to the "Markdown to Textual Data Extractor" n8n node, which leverages a Google Gemini LLM to convert the complex markdown into clean, raw textual data, removing links, scripts, and other non-text elements. The output is also sent to a notification webhook.


4. Parallel Analysis: The clean text is simultaneously analyzed by two specialized Information Extractor n8n nodes:

1. Local File Storage: Finally, the structured results from both extraction paths are written to the local disk using the "Write the topics file to disk" and "Write the trends file to disk" n8n node components.

1. Import: Copy the provided n8n workflow JSON and paste it into your n8n instance.


2. Credentials Setup:

1. Configuration: Update the "Set URL and Bright Data Zone" n8n node with the desired url and zone parameters specific to your scraping needs. This n8n node controls the target.


2. Webhooks: Update all instances of the HTTP Request n8n node (Webhooks) with your desired notification endpoint (e.g., a monitoring service URL). Ensure the webhook URL in this n8n workflow is correct.


3. Local File Paths (Self-Hosted n8n): If self-hosting, ensure the file paths in the Write the topics file to disk and Write the trends file to disk n8n nodes (e.g., d:\topics.json) are accessible and valid within the n8n execution environment.

When clicking ‘Test workflow’ (Manual Trigger n8n trigger): Starts the n8n workflow execution upon manual click, ideal for testing and development of n8n templates.
Set URL and Bright Data Zone (Set n8n node): Defines two key parameters, url (the target website) and zone (the Bright Data Web Unlocker configuration), used downstream in this n8n workflow.
Perform Bright Data Web Request (HTTP Request n8n node): Sends a POST request to Bright Data's API, configured with HTTP Header Authentication to securely retrieve webpage content. Key setting: data_format is set to markdown.
Markdown to Textual Data Extractor (Chain LLM n8n node): Acts as an intermediate cleaning step. Uses Google Gemini to convert raw markdown data into clean text based on a system prompt to output textual data only.
Topic Extractor with the structured response (Information Extractor n8n node): Performs advanced topic modeling using Google Gemini. Key Configuration: Uses a custom JSON schema to enforce structured output for topic, score, summary, and keywords.
Trends by location and category with the structured response (Information Extractor n8n node): Executes complex trend analysis. Key Configuration: Uses a detailed JSON schema to cluster emerging trends by location and category, ensuring the output is perfectly structured.
Create a binary file for topics / Create a binary data for tends (Function n8n node): Converts the structured JSON output from the AI extraction steps into a base64-encoded binary format, preparing it to be written to disk by the next n8n node.
Write the topics file to disk / Write the trends file to disk (Read/Write File n8n node): Saves the structured analysis results as local JSON files (topics.json and trends.json) within the n8n execution environment.