AI-Powered Web Scraper using Gemini and ScrapeNinja - n8n Workflow

• Technical Marketers needing reliable data extraction tools.


• Developers looking to automate web data gathering without writing manual parsing scripts.


• Data Scientists requiring structured inputs from difficult-to-scrape sites.


• Anyone seeking advanced n8n templates that combine web scraping and large language models (LLMs).

This advanced n8n workflow solves the common problem of creating and maintaining brittle web scraper scripts. Instead of manually inspecting element selectors and writing complex Cheerio or Puppeteer code, this automation leverages the power of Google Gemini. The n8n node sequence first scrapes the target page, then cleans the raw HTML. This cleaned HTML is fed into the LLM, which is prompted to generate the precise JavaScript function needed to extract specific data fields (like titles, URLs, and scores).

This method makes the n8n workflow highly adaptable. If the target site's layout changes slightly, the AI can often generate updated extraction code on the fly. This sophisticated n8n template demonstrates how to execute dynamic, AI-generated code safely within a secure environment, providing structured JSON output from any webpage. It’s an essential tool for any serious n8n user engaging in web data tasks.

This n8n workflow operates in five distinct stages, beginning with a simple n8n trigger:

1. Manual Trigger: The automation starts via the Generate custom web scraper manual n8n trigger, initiating the data flow.


2. Initial Scrape: The first ScrapeNinja n8n node accesses the target URL (configured here as https://news.ycombinator.com/) and fetches the raw HTML content.


3. HTML Preparation: The Cleanup HTML n8n node takes the raw output and uses the dedicated cleanup-html operation to standardize and simplify the code, making it easier for the AI to analyze.


4. AI Code Generation: The cleaned HTML is passed to the Generate JS eval code via LLM n8n node, which is powered by the connected Google Gemini Chat Model. A specific prompt instructs the AI to generate a complete Cheerio.js function designed to extract news items and return them in a structured JSON array format. The prompt strictly requires the response to start with function extract(html, cheerioInstance) { ... }.


5. Execution and Extraction: Finally, the Eval generated code to extract data n8n node executes the extraction function received from the LLM against the original cleaned HTML. This performs the extract-custom operation, resulting in a structured, actionable JSON data set, completing this powerful n8n workflow.

To deploy this powerful n8n workflow template, follow these steps:

1. Import: Copy the provided n8n workflow JSON and paste it directly into your n8n canvas.


2. Credentials Setup: You must configure two primary credentials for the n8n node setup:

1. Configuration: Update the ScrapeNinja n8n node parameters to reflect the specific URL you wish to scrape.


2. Testing: Use the manual Generate custom web scraper n8n trigger to test the entire flow and verify that the Eval generated code to extract data n8n node successfully outputs the structured JSON data.

This n8n workflow uses several critical n8n node types to achieve its goal:

Generate custom web scraper (Manual Trigger): The starting n8n trigger point for manual execution.
ScrapeNinja (Initial Scrape): Function: Fetches the raw HTML content from the specified URL (https://news.ycombinator.com/).
Cleanup HTML (ScrapeNinja): Function: Cleans and simplifies the raw HTML using the cleanup-html operation, preparing it for LLM analysis.
Google Gemini Chat Model (Language Model): Function: Provides the AI engine (models/gemini-exp-1206) necessary for generating dynamic JavaScript code.
Generate JS eval code via LLM (Chain LLM): Function: Sends the cleaned HTML and a highly specific prompt to the Gemini LLM. It dictates that the AI must return a runnable cheerio.js function for parsing the news items. This is a crucial n8n node for the workflow's adaptability.
Eval generated code to extract data (ScrapeNinja): Function: Executes the AI-generated JavaScript function against the cleaned HTML content using the extract-custom operation. This final n8n node yields the desired structured JSON output.