Ultimate Selenium and GPT-4o Vision Web Scraper - n8n Workflow

• Organizations needing to extract complex data from websites that require login or employ strict anti-bot measures.


• Developers looking for robust n8n templates for enterprise-grade web scraping.


• Data analysts requiring structured information extracted from visually rendered web pages.


• n8n users integrating AI (specifically GPT-4o) into their data pipelines.

This is the ultimate scraping solution built entirely within an n8n workflow. Traditional web scraping often fails against modern anti-bot measures and complex JavaScript rendering. This n8n workflow bypasses these issues by using a dedicated Selenium Chrome container configured for stealth. It is triggered by an n8n trigger (Webhook) and can handle two main scenarios: searching for a relevant URL via Google or scraping a predefined Target Url. Crucially, it supports injecting session cookies for authenticated scraping (e.g., pages requiring login). Instead of fragile CSS selectors, the workflow takes a screenshot and uses the OpenAI GPT-4o Vision n8n node to visually analyze the page and extract the requested data fields, making the process highly resilient and accurate. This complex n8n workflow ensures high success rates even on challenging targets.

The n8n workflow begins with a POST request received by the Webhook n8n trigger.

1. Input Handling: The workflow first determines the scraping strategy. If only a subject and domain are supplied, the If Target Url n8n node directs execution to perform a Google Search (Google search Query) scoped to the provided domain to find the best page URL (Extract First Url Match). An AI Information Extractor then validates the best URL.


2. Session Creation: A Selenium Chrome session is created (Create Selenium Session) via HTTP requests to a local Selenium Docker service (selenium_chrome:4444). The session is configured with anti-detection arguments.


3. Authentication Path (Conditional): An If n8n node checks for incoming cookies. If present, a Code n8n node processes the cookies (standardizing sameSite values), and the Inject Cookie n8n node uploads them to the Selenium browser session for authenticated access.


4. Navigation: The Selenium browser navigates to the final target URL (either direct or extracted via Google search).


5. Visual Scraping: The workflow instructs the Selenium session to take a screenshot (Get ScreenShot). This image is converted to binary data by the Convert to File n8n node.


6. AI Analysis & Block Detection: The powerful GPT-4o Vision OpenAI n8n node receives the image and a specific prompt. It analyzes the screenshot to determine if the site is blocked (returning 'BLOCK') or to extract the required data. This provides a resilient alternative to standard HTML parsing.


7. Data Structuring: If the page is not blocked, the raw text output from GPT-4o is passed to the LangChain Information Extractor n8n node, which structures the data precisely according to the user's defined Target data schema.


8. Conclusion: The results are returned via a successful Webhook response (Success), and multiple Delete Session HTTP Request n8n nodes ensure that the Selenium resource is always gracefully terminated, regardless of success or failure throughout the n8n workflow execution.

To deploy this advanced n8n workflow, several external dependencies are required in addition to importing the JSON:

1. Import the n8n template: Copy the provided JSON and import it into your n8n instance.


2. Configure Credentials: Set up credentials for the OpenAI API connection used by the AI n8n node configurations.


3. Selenium Setup (Crucial): This n8n workflow requires an external Selenium container accessible via http://selenium_chrome:4444. You must deploy the associated Docker Compose file (refer to the GitHub project linked in the sticky note). Ensure your n8n container can communicate with the Selenium container.


4. Proxy Configuration (Optional but Recommended): If using a residential proxy, configure it within the Selenium Docker setup and potentially add the proxy arguments (--proxy-server=address:port) to the Create Selenium Session n8n node's JSON body. Note that Selenium often requires IP whitelisting for proxy authentication.


5. Initial Test: Trigger the Webhook n8n trigger using a POST request tool (like cURL or Postman) using the provided example format, ensuring you substitute the webhook ID with the actual ID from the Webhook n8n node.

This complex n8n workflow utilizes core logic, HTTP requests for browser control, and advanced AI integration:

Webhook (n8n trigger): The starting point for the n8n workflow, configured to receive POST requests carrying all necessary scraping parameters (URL, subject, cookies, target data).
Create Selenium Session (HTTP Request n8n node): Initiates a headless Chrome session. Key settings include anti-detection arguments and a connection timeout.
Code (n8n node): A custom scripting n8n node used specifically to normalize incoming cookie sameSite attributes to values accepted by Selenium ('Strict', 'Lax', 'None').
Inject Cookie (HTTP Request n8n node): Posts the structured cookie JSON data to the active Selenium session to bypass login pages.
If nodes (If Target Url, If, etc.): These If n8n nodes manage flow control, checking for the presence of input cookies or a target URL to ensure the n8n workflow follows the correct path.
Get ScreenShot (HTTP Request n8n node): Instructs the Selenium browser to capture the fully rendered webpage as a base64 encoded image.
Convert to File (n8n node): Converts the base64 screenshot output into a binary format suitable for OpenAI's Vision API.
OpenAI / OpenAI1 (OpenAI n8n node): Performs image analysis using GPT-4o Vision. The prompt is highly specialized, asking the AI to extract specific data from the screenshot or return a 'BLOCK' signal if a WAF or anti-bot screen is detected.
Information Extractor (LangChain n8n node): This highly specialized AI n8n node takes the unstructured text summary from GPT-4o and strictly formats it into the specific, named fields requested by the user's Target data input, ensuring structured and usable JSON output. This is a critical piece of the n8n template for data processing.
Delete Session (HTTP Request n8n node): Multiple cleanup nodes, crucial for resource management, ensuring the Selenium browser session is terminated after the n8n workflow completes, even on error paths.