AI Agent Driven ProductHunt Data Extraction and Search using Bright Data & Gemini - n8n Workflow

• Market researchers and analysts needing competitive intelligence.


• Developers and users interested in leveraging powerful AI Agents within an n8n workflow.


• Anyone who needs to extract and structure large amounts of data from specific websites like ProductHunt.


• Technical users running a self-hosted n8n environment who can install community n8n node integrations.

Extracting clean, actionable data from complex websites often requires more than simple scraping; it demands intelligence to interpret context. This powerful n8n workflow solves this challenge by integrating the robust scraping capabilities of Bright Data’s Master Control Panel (MCP) with the sophisticated reasoning of the Google Gemini AI Agent.

The automation starts by defining specific input parameters (like the ProductHunt category and a search query). The core of this n8n template is the AI Agent, which dynamically decides whether to use the Bright Data tool for extracting content (via the scrapeasmarkdown n8n node) or perform a general Google search (via the search_engine n8n node) to fulfill the user-defined goal. The resulting data, even if initially unstructured, is then processed by a secondary LLM chain to enforce a strict JSON schema, making it immediately useful for downstream systems like Google Sheets. This n8n workflow demonstrates how to build highly adaptive, data-intensive solutions.

The entire process is initiated by the Manual n8n trigger, making it easy to test and execute.

1. Initialization and Input Setting: The workflow first calls the MCP Client to list tools, followed by the ‘Set the Input Fields’ n8n node which defines all necessary inputs (base URL, target category, search query, and webhook URL).


2. AI Agent Execution: The ‘AI Agent’ n8n node, powered by the Google Gemini Chat Model, receives an objective (e.g., “Perform a Product Hunt data extract”). The Agent dynamically chooses between the ‘MCP Client for Google Search’ and the ‘MCP Client for Markdown Data Extract’ n8n node based on the prompt’s needs.


3. Raw Data Processing: The full output generated by the AI Agent is then captured. This raw output is simultaneously archived to a designated Google Sheet, converted to a binary format and saved to a local disk (using the Function and Read/Write File n8n node), and immediately sent via an HTTP Request n8n node (Initiate a Webhook Notification).


4. Structured Refinement: To transform the verbose AI output into clean, structured data, the ‘Structured Data Extractor’ LLM Chain takes the raw output. It is configured with a specific JSON schema using the ‘Structured Output Parser’ n8n node.


5. Final Output: The final structured data (containing links and descriptions) is appended to a second, designated Google Sheet using a dedicated Google Sheets n8n node, completing this complex n8n workflow.

To install this robust n8n workflow, follow these steps:

1. Import: Copy the provided JSON code and paste it into your n8n instance using the 'New' -> 'Import from JSON' option.


2. Community Node Requirement: This n8n template relies on the community n8n-nodes-mcp node for Bright Data functionality. You must install this in your n8n environment if it is not already present. (Note: This is often required for self-hosted instances).


3. Credentials Setup:

1. Configuration: Update the values in the ‘Set the Input Fields’ n8n node (especially category, search, and webhookurl) to match your specific extraction goals before executing the n8n trigger.

This n8n workflow leverages several specialized n8n node types:

When clicking ‘Execute workflow’ (Manual Trigger): The standard starting n8n trigger for immediate manual execution.
Set the Input Fields (Set n8n node): Crucial for defining runtime variables like base_url, category (e.g., resumes), and the target search query.
MCP Client for Google Search / Markdown Data Extract (MCP Client Tool n8n node): These nodes serve as the tools for the AI Agent, allowing it to execute external web operations (Google Search and scraping Product Hunt data in Markdown format) using Bright Data.
AI Agent (Langchain Agent n8n node): The brain of the automation. It takes the user prompt and utilizes the connected Google Gemini model and Bright Data tools to produce a comprehensive output.
Structured Data Extractor (Langchain ChainLlm n8n node): Used post-AI Agent to refine the output. It uses a specific prompt to extract key fields (links, description, keywords) and enforces structure using the Output Parser.
Google Sheets (Google Sheets n8n node): Used twice in this n8n workflow to persist the data—once for the raw AI agent output and once for the cleaned, structured data.