Multi-Engine Search & Structured Data Extraction with Bright Data and Gemini - n8n Workflow

Data Analysts needing clean, structured search results from multiple sources.
Technical users running self-hosted n8n who require advanced web scraping capabilities.
Automation Specialists looking for a robust AI agent n8n node implementation.
Developers who need to automate complex web data acquisition tasks.

Acquiring accurate, structured data from major search engines (Google, Bing, Yandex) can be challenging due to scraping complexity and the need to process raw HTML results. This comprehensive n8n workflow solves this by utilizing the Bright Data Managed Collecting Product (MCP) client alongside powerful AI capabilities provided by Google Gemini.

This n8n template uses an AI Agent architecture to intelligently select the correct search engine tool based on user input, executes the query, and then processes the unstructured output using a second Gemini-powered LLM. The resulting clean, human-readable data is saved locally and sent via a webhook, ensuring a reliable and structured output for subsequent steps in your broader data pipeline. This particular n8n workflow requires a self-hosted installation due to its use of community n8n node components.

This automation is initiated via a manual n8n trigger, making it perfect for testing or scheduled runs when wrapped in an appropriate n8n node schedule.

1. Input Definition: The initial step, following the n8n trigger, uses the 'Set the Input Fields' n8n node to define the search query (e.g., 'Bright Data'), the specific action (e.g., 'Perform Bing search'), and a webhooknotificationurl.


2. AI Agent Execution: The data flows into the 'Bright Data Search AI Agent' n8n node. This agent, powered by the 'Google Gemini Chat Model for Search Agent' language model, interprets the desired action and automatically selects the correct Bright Data MCP tool (Google, Bing, or Yandex).


3. Search & Scrape: The selected Bright Data MCP n8n node executes the search and returns the raw results.


4. Data Cleaning: The raw output is passed to the 'Human Readable Data Extractor' n8n node. This second AI component, leveraging a dedicated Google Gemini model, cleans the search results, removes irrelevant links or HTML tags, and extracts the core information into a structured JSON attribute called search_response.


5. Output & Storage: The final cleaned data is simultaneously converted to a binary format and written to a specified local file (d:\Scraped-Search-Results.json) using the File Read/Write n8n node. Concurrently, the extracted data is sent to the defined webhook notification URL for immediate use or integration.

To install this n8n workflow, follow these steps:

1. Prerequisites: This n8n template requires a self-hosted instance of n8n due to the community nature of the Bright Data MCP Client nodes used.


2. Import Workflow: Copy the provided JSON data and paste it directly into your n8n canvas using the 'New' > 'Import from JSON' option.


3. Bright Data Credentials: Configure credentials for the Bright Data MCP Client API on the relevant 'MCP Client' n8n node components.


4. Google Gemini Credentials: Ensure you have configured your Google Gemini (PaLM) API Key credentials for both the 'Search Agent' and the 'Human Readable Data Extractor' n8n nodes.


5. Configuration: Edit the 'Set the Input Fields' n8n node:

1. Activate: Save the n8n workflow and run a test execution.

Manual Trigger n8n trigger: Initiates the n8n workflow manually for testing or single-run execution.
Set the Input Fields (n8n node): Essential configuration node. Defines the search query, the target search engine (action), and the notification webhook URL.
Bright Data Search AI Agent (@n8n/n8n-nodes-langchain.agent): The core intelligence of this n8n workflow. It uses the Gemini LLM to decide which of the three configured MCP tools to execute based on the action input.
MCP Client Tools (Google, Bing, Yandex Search): Custom n8n node implementations for interacting with Bright Data's search APIs. They scrape the results based on the query provided by the AI Agent.
Google Gemini Chat Models (n8n node): Used as the underlying LLM for both the Search Agent and the Data Extractor, employing the models/gemini-2.0-flash-exp model.
Human Readable Data Extractor (@n8n/n8n-nodes-langchain.informationExtractor): Cleans the raw search results using AI prompting, ensuring the final output is structured and clean.
Write the search result to disk (n8n node): Writes the final, cleaned JSON data to a local disk path, crucial for self-hosted n8n setups requiring local storage.
Webhook for clean data extractor (HTTP Request n8n node): Sends the final structured search_response to an external URL for notification or subsequent integration.