Etsy Data Mining and Structured Extraction with Bright Data and Google Gemini - n8n Workflow

Markdown-formatted. Market analysts needing to track product trends on e-commerce platforms like Etsy. Developers looking for advanced examples of web scraping paired with AI data extraction within n8n. Users who require a robust, paginated web scraping solution using the n8n platform. Anyone interested in using the n8n templates ecosystem to build data pipelines involving Bright Data and Google Gemini.

This sophisticated n8n workflow solves the challenge of scraping dynamic, paginated e-commerce sites like Etsy. Standard scraping methods often fail due to anti-bot measures, which is mitigated here by utilizing Bright Data’s Web Unlocker. The crucial innovation of this n8n workflow lies in its use of Large Language Models (LLMs) to handle data extraction. Instead of relying on brittle CSS selectors, the workflow uses the Google Gemini n8n node to reliably identify pagination links and extract detailed product information based on a predefined JSON schema. This ensures the output is clean, structured, and ready for further analysis, making this one of the most powerful n8n templates for data intelligence.

This n8n workflow initiates with a Manual n8n trigger and follows a three-stage process: Initialization, Pagination Analysis, and Iterative Extraction.

1. Initialization: The n8n trigger starts the process. The 'Set Esty Search Query' n8n node defines the initial search URL and the Bright Data zone (web_unlocker1). The 'Perform Esty Web Request' n8n node scrapes the first page using Bright Data.

1. Pagination Analysis (LLM-Powered): The raw HTML data is passed to the 'Extract Paginated Resultset' n8n node. This Langchain Information Extractor node, powered by the Google Gemini Chat Model, analyzes the page content and extracts all available pagination URLs into a clean array, overcoming typical scraping hurdles.

1. Iterative Extraction & Output: The 'Split Out' and 'Loop Over Items' n8n nodes iterate through the list of extracted pagination URLs.

1. Final Actions: The extracted data is sent to a webhook for immediate notification ('Initiate a Webhook Notification') and is simultaneously converted to binary format and saved as a JSON file locally using the 'Write the scraped content to disk' n8n node, demonstrating robust data handling capabilities in this n8n template.

Markdown-formatted.

1. Import: Copy the full n8n workflow JSON provided and paste it into your n8n instance using the 'New' -> 'Import from JSON' option.


2. Bright Data Credentials: You must create an HTTP Header Auth credential in n8n for Bright Data, providing your API key, which is used by the 'Perform Esty Web Request' n8n node.


3. Google Gemini Credentials: Configure a Google Gemini (PaLM) API credential and link it to both 'Google Gemini Chat Model' n8n nodes.


4. URL Update: In the 'Set Esty Search Query' n8n node, update the url value to your desired Etsy search query.


5. Webhook Configuration: Update the URL in the 'Initiate a Webhook Notification' n8n node (currently set to a placeholder webhook.site URL) to your actual notification endpoint.


6. Run: After connecting all credentials, click the 'Test workflow' n8n trigger or activate the full n8n workflow.

Markdown-formatted. When clicking ‘Test workflow’ (Manual Trigger): The starting n8n trigger for executing this n8n workflow on demand. Set Esty Search Query (Set n8n node): Defines global variables for the scrape operation, specifically the target Etsy url (https://www.etsy.com/search?q=wall+art+for+mum...) and the Bright Data zone. Perform Esty Web Request (HTTP Request n8n node): Executes the initial web scrape request through the Bright Data API endpoint. Key configuration includes using POST method and passing the target URL and zone in the body parameters. Google Gemini Chat Model (Langchain LLM): Provides the language model connectivity for structured output generation. It uses the models/gemini-2.0-flash-exp model, linking directly to the Information Extractor n8n node. Extract Paginated Resultset (Information Extractor n8n node): This crucial n8n node uses Gemini to analyze the scraped HTML and extract all pagination links, conforming to a strict PagedResultSetSchema for reliable looping. Loop Over Items (Split In Batches n8n node): Manages the iteration process, handling each extracted pagination URL sequentially to ensure every page is scraped. Perform Esty web request over the loop (HTTP Request n8n node): Executes the actual scrape for each paginated URL within the loop, leveraging Bright Data's Web Unlocker. Extract Item List with the Product Info (Information Extractor n8n node): Uses Gemini again, this time with a product-focused JSON schema, to pull detailed information like price, brand, and image URL from the individual product listings on each scraped page. Initiate a Webhook Notification for the extracted data (HTTP Request n8n node): Sends the final, cleaned JSON data summary to an external monitoring service or application. Write the scraped content to disk (Read/Write File n8n node): Saves the extracted data locally. The file name is dynamically generated using the page number (Esty-Scraped-Content-{{ $('Loop Over Items').item.json.page_number }}.json).