Indeed Company Data Scraper and AI Summarization Automation - n8n Workflow

• Recruitment specialists needing automated company profiles.


• Market researchers extracting competitive intelligence.


• Developers looking for advanced web scraping n8n templates.


• Users seeking complex AI and web scraping integration examples in an n8n workflow.

This comprehensive n8n workflow demonstrates high-level integration between external scraping services (Bright Data), database management (Airtable), and cutting-edge AI processing (Google Gemini via LangChain nodes). The core problem solved is transforming raw, large-scale web data into actionable, summarized insights.

Starting with an n8n trigger, the process efficiently reads a list of company links from an Airtable base. It uses the Bright Data Web Unlocker product via an HTTP Request n8n node for robust scraping of Indeed pages, ensuring high success rates. Subsequently, it leverages several specialized LangChain n8n nodes (LLM Chain, Summarization Chain, and AI Agent) powered by Google Gemini to extract and condense the complex markdown results. This entire n8n workflow is a perfect example of how modern automation overcomes complex data extraction and processing challenges using sophisticated n8n node configurations.

1. Start: The n8n workflow is initiated manually using the "When clicking ‘Test workflow’" n8n trigger.


2. Configuration & Data Retrieval: A Set n8n node defines the Bright Data zone (web_unlocker1). The Airtable n8n node fetches a batch of Indeed links (company pages) that need processing from the specified base.


3. Iteration & Control: The Loop Over Items n8n node processes each company link individually. A Wait n8n node introduces a 10-second pause to respect rate limits, and an If n8n node ensures that only records with non-empty links proceed.


4. Web Scraping: The Perform Indeed Web Request n8n node sends a request to the Bright Data Web Unlocker API, dynamically passing the Indeed company URL retrieved from Airtable. The scraper returns the data in markdown format.


5. Markdown to Text: The raw markdown output is first passed to the Markdown to Textual Data Extractor LLM Chain n8n node, which uses a dedicated Google Gemini Chat Model n8n node to convert the scraped content into clean textual data.


6. Summarization: The clean text is fed into the Indeed Summarizer Summarization Chain n8n node. This uses a second instance of the Google Gemini Chat Model For Summarization n8n node to generate a concise summary of the company profile.


7. AI Agent Action: The generated summary is handled by the Indeed Expert AI Agent n8n node. This powerful agent, using a third Google Gemini Chat Model for AI Agent n8n node, executes a defined action: utilizing the Webhook HTTP Request n8n node tool to format the final summarized response and push the results to a specified external webhook endpoint. The loop then continues to process the next company link, ensuring full automation of the data pipeline.

1. Import: Import this n8n workflow JSON into your n8n instance.


2. Airtable Credentials: Update the Airtable n8n node with your Personal Access Token. Ensure the Base and Table names match the source of your Indeed links (as noted in the Sticky Note: Base 'Indeed', Table 'Table 1').


3. Bright Data Credentials: Configure the HTTP Header Authentication credentials used by the Perform Indeed Web Request n8n node with your Bright Data API token.


4. Google Gemini Credentials: Configure the Google Gemini (PaLM) API credentials for all three AI-related n8n nodes.


5. Webhook Configuration: Update the URL in the Webhook HTTP Request n8n node (the tool used by the AI Agent) and the Initiate a Webhook Notification... n8n node to your desired notification endpoints.

Manual Trigger: The necessary starting n8n trigger to manually launch the data scraping process.
Airtable n8n node: Fetches the queue of company links from your configured Airtable base to initiate the looping process.
Perform Indeed Web Request n8n node (HTTP Request): Configured to interface with the Bright Data Web Unlocker, dynamically scraping the Indeed URL specified in the Airtable record.
Google Gemini Chat Model n8n node (x3): Provides the underlying large language model (models/gemini-2.0-flash-exp) for all complex AI processing steps in this n8n workflow.
Markdown to Textual Data Extractor n8n node (LangChain LLM Chain): Utilizes the LLM to clean and transform the raw markdown output from the scraper into structured textual data.
Indeed Summarizer n8n node (LangChain Summarization Chain): A specialized n8n node that generates a concise summary of the cleaned company data, focusing on key insights.

• Indeed Expert AI Agent n8n node (LangChain Agent): The final processing step. This n8n node directs the LLM to use the Webhook HTTP Request tool to send the summarized data in a defined format.