Data Extraction and Daily Database Sync for GitHub Trending - n8n Workflow

Markdown-formatted:

Data engineers or developers tracking open-source trends.
Users who need reliable, scheduled web scraping delivered directly to a database.
Anyone seeking an advanced example of using the n8n AI Agent node with external tools (like databases).
System administrators looking for robust n8n templates for data pipeline creation.

Tracking GitHub trending repositories is crucial for understanding current developer interests and emerging technologies. However, manually gathering and structuring this data is time-consuming. This specialized n8n workflow solves this problem by automating the entire pipeline.

It leverages scheduled n8n triggers to run the collection process daily, weekly, and monthly. It uses the Firecrawl n8n node for high-fidelity web scraping and, most critically, utilizes an n8n AI Agent (powered by GPT-4o-mini) to reliably transform unstructured Markdown scrape output into clean, structured JSON data. This structured data is then inserted directly into your Supabase database, providing a centralized and searchable history of GitHub's hottest projects. This is one of the most comprehensive n8n templates available for combining scraping, AI extraction, and database management.

Markdown-formatted:

1. Scheduled Activation: The n8n workflow initiates via one of three dedicated Schedule n8n trigger nodes: daily, weekly, or monthly.


2. Variable Assignment: The flow immediately passes through specific Set nodes (B1, B2, B3) that assign the appropriate timeframe (daily, weekly, or monthly) to a workflow variable. The consolidated B_ALL Set node prepares this variable for subsequent steps.


3. Data Scraping (Firecrawl n8n node): The Firecrawl n8n node executes a scrape of the GitHub trending page, dynamically using the variable assigned in Step 2 (e.g., https://github.com/trending?since=daily).


4. AI Extraction Orchestration: The core logic resides in the AI Agent n8n node. This node receives the scraped Markdown content and, using a highly precise system prompt, instructs the LLM (powered by the OpenAI Chat Model n8n node) to extract structured fields like project name, URL, description, language, and the numerical star count.


5. Database Insertion: The AI Agent is configured with the 'Create a row in Supabase' tool. Once the LLM successfully extracts the data into the defined JSON array format, the AI Agent orchestrates the execution of the Supabase n8n node, iterating through each project item and inserting the fully structured record into the githubtrending table, including the required type field.

Markdown-formatted:

1. Import the n8n workflow: Copy the provided JSON data and paste it into your n8n instance using the 'New' -> 'Import from JSON' option.


2. Configure Credentials: You will need to set up credentials for the following services:

1. Database Setup: Crucially, run the SQL script provided in the Sticky Note within your Supabase SQL Editor to create the githubtrending table with the necessary schema:

CREATE TABLE public.githubtrending (
      id bigint GENERATED ALWAYS AS IDENTITY NOT NULL,
      createdat timestamp with time zone NOT NULL DEFAULT now(),
      datadate date DEFAULT now(),
      url text,
      projectid text,
      projectdesc text,
      codelanguage text,
      stars bigint DEFAULT '0'::bigint,
      type text,
      CONSTRAINT githubtrendingpkey PRIMARY KEY (id)
    );

1. Activation: Once credentials and the table are ready, activate the n8n workflow to begin scheduled execution.

Markdown-formatted:

Schedule Trigger (daily, weekly, monthly): The primary n8n trigger nodes. They handle the execution frequency, ensuring the collection process runs automatically at the specified intervals.
Set (B1, B2, B3, B_ALL): Core Logic flow control. These n8n node instances define and pass the required timeframe variable (daily, weekly, or monthly) downstream, which is essential for determining the correct GitHub URL parameter.
Firecrawl: This dedicated scraping n8n node fetches the content from the GitHub Trending page.
Function: Reliable scraping, specifically targeting the main content and project listings (div[data-hpc]).
Key Configuration: URL is dynamic: =https://github.com/trending?since={{ $json.type }}.
AI Agent: The orchestrator of the data cleaning process.
Function: Uses the linked OpenAI Chat Model (GPT-4o-mini) and the Supabase tool to extract data from the scraped Markdown text and perform the insertion.
Key Configuration: Includes a strict System Message dictating the required JSON output format and field names (name, url, description, language, stars).
OpenAI Chat Model: Provides the necessary Large Language Model (LLM) capability for the AI Agent.
Function: Executes the prompt instructions given by the AI Agent to perform precise data structuring and extraction. Uses gpt-4o-mini for cost-effectiveness and speed.
Create a row in Supabase: This n8n node acts as a tool for the AI Agent.
Function: Inserts the extracted, structured data directly into the githubtrending table. Field mapping is configured using AI-driven output variables ($fromAI).