Vision-Based AI Scraper for Structured Data Extraction using Gemini and Google Sheets - n8n Workflow

Web scraping specialists needing dynamic, vision-based data extraction.
E-commerce analysts monitoring competitors' product data and pricing.
Users looking for advanced n8n templates integrating multimodal AI (Gemini).
Technical users who want to implement a robust, hybrid scraping n8n node approach (screenshot + HTML fallback).

This sophisticated n8n workflow solves the challenge of extracting structured data from complex websites where traditional selectors often fail. By using the vision capabilities of the Gemini AI model, the n8n workflow first analyzes a full-page screenshot (captured via ScrapingBee) to identify product details, prices, and promotions. If the visual extraction is incomplete, the AI agent dynamically calls a secondary n8n tool to retrieve the raw HTML. This hybrid approach, orchestrated entirely within this n8n workflow, ensures maximum data accuracy and reliability. The results are then neatly formatted using a structured output parser and appended to Google Sheets, providing a powerful, automated data pipeline. This entire n8n setup acts as a high-performance scraping agent.

1. Trigger & Data Input: The n8n workflow starts manually (via the test n8n trigger, which can be replaced). It fetches a list of target URLs from a Google Sheets document.


2. URL Preparation: A Set n8n node prepares the URL data for the next steps.


3. Screenshot Capture: The ScrapingBee n8n node captures a full-page screenshot of the target URL, passing the binary image data to the AI agent.


4. Vision-Based Agent Execution: The core Vision-based Scraping Agent n8n node takes the screenshot and URL. It uses the Google Gemini Chat Model (Gemini 1.5 Pro) to analyze the image based on specific instructions (extracting titles, prices, etc.).


5. Conditional Fallback: If the Gemini AI cannot fully extract the required data from the visual input, it calls the nested HTML-based Scraping Tool.

1. Structured Parsing: The AI agent returns the extracted data, which is enforced into a standardized JSON array structure by the Structured Output Parser n8n node.


2. Data Persistence: The Split Out n8n node separates the JSON array into individual data items, and the final Google Sheets n8n node appends each item as a new row to the specified "Results" sheet. This completes the robust data extraction cycle managed by this powerful n8n workflow.

This n8n template requires several credentials and configurations:

1. Import: Import this n8n workflow JSON into your n8n instance.


2. Google Sheets Setup: Configure a Google Sheets Service Account credential. Update the Google Sheets - Get list of URLs n8n node and the Google Sheets - Create Rows n8n node to point to your specific spreadsheet ID and sheet names ("List of URLs" and "Results").


3. ScrapingBee API: Replace scrapingbeeapikey> placeholders in both ScrapingBee HTTP Request n8n nodes with your actual API key.


4. Google Gemini (PaLM) API: Set up the Google Gemini API credential for the Google Gemini Chat Model n8n node.


5. Workflow ID (Self-Reference): In the HTML-based Scraping Tool n8n node, ensure the workflowId parameter correctly references the ID of this current n8n workflow, as it calls itself recursively as a tool. This is a critical step for the proper functioning of the entire n8n workflow.

When clicking ‘Test workflow’ (n8n trigger): Manually initiates the n8n workflow execution.
Google Sheets - Get list of URLs (n8n node): Retrieves the list of URLs from the source Google Sheet for processing.
ScrapingBee - Get page screenshot (HTTP Request n8n node): Uses the ScrapingBee API to capture a screenshotfullpage (set to true) of the target URL, essential for the vision-based analysis in the downstream n8n node.
Vision-based Scraping Agent (LangChain Agent n8n node): The core intelligence. It uses the screenshot and, conditionally, the HTML (via the embedded tool) to extract data according to the detailed System Prompt instructions.
Google Gemini Chat Model (n8n node): The LLM powering the agent, configured to use models/gemini-1.5-pro-latest for superior visual processing capabilities within the n8n workflow.
HTML-based Scraping Tool (Workflow Tool n8n node): A crucial fallback mechanism. It defines a tool the AI agent can call if visual data is insufficient.
Structured Output Parser (n8n node): Defines the strict JSON schema required for the final output (e.g., producttitle, productprice). This ensures the n8n workflow provides clean, standardized data.
ScrapingBee- Get page HTML (HTTP Request n8n node): Retrieves the raw HTML source of the page, used only when the agent specifically calls the HTML scraping tool.
HTML to Markdown (n8n node): Converts bulky HTML into more efficient Markdown, reducing token count before the data is processed by the AI agent, optimizing the entire n8n workflow cost.
Google Sheets - Create Rows (n8n node): Appends the final, structured, AI-parsed data into the designated "Results" sheet.