Automated News Aggregation and Database Sync - n8n Workflow

Markdown

Managing and normalizing data from multiple news APIs can be complex due to varying response schemas. This comprehensive n8n template provides a plug-and-play solution for building a centralized news database. The system is designed with four parallel, scheduled flows, ensuring reliable extraction of top headlines and category-specific news from NewsAPI, Mediastack, and CurrentsAPI. Each stream utilizes a dedicated n8n node structure to perform meticulous data normalization before storing the articles consistently in NocoDB. This robust n8n workflow saves significant development time, offering an immediate solution for content analysis pipelines, research tools, or automated editorial queues. Utilizing the power of n8n, developers and analysts can ensure they always have fresh, standardized data.

Markdown
This n8n workflow operates using four independent, scheduled branches, ensuring comprehensive and consistent data collection:

1. Scheduling & Configuration (n8n Trigger): Four separate Schedule Trigger n8n node setups initiate the extraction processes at defined intervals (daily or bi-daily).


2. Configuration and Itemization: For category-based providers (NewsAPI Categories, Mediastack), a Set n8n node defines the desired categories (e.g., business, technology). A subsequent Code n8n node transforms this category array into individual items.


3. Looping and API Calls: The categories are processed in parallel using a Split In Batches n8n node, triggering multiple HTTP Request n8n nodes tailored for each provider (NewsAPI, Mediastack, CurrentsAPI).


4. Data Extraction and Splitting: The API responses are fed into Split Out n8n node to separate the core list of articles (articles or data/news) into individual items ready for processing.


5. Data Normalization (Set n8n node): Crucially, dedicated Set n8n node instances map the unique field names (like urlToImage, description, source.name) from each aggregator's response into a single, standardized schema (e.g., title, aggregator, publisher, summary).


6. Database Insertion: The normalized data is then inserted into the target NocoDB table using the NocoDB n8n node, completing the automation cycle.

Markdown
To successfully deploy and run this n8n workflow, you must complete the following critical setup steps:

1. Import the n8n Template

1. Copy the provided n8n workflow JSON.


2. In your n8n instance, click 'New' and then 'Import from JSON'.


3. Paste the JSON and save the n8n workflow.

2. Configure API Keys

3. Set Up NocoDB Credentials

1. Go to Credentials in n8n.


2. Locate the existing 'NocoDB Token account' credential used by the workflow.


3. Update the API Token and Base URL to match your NocoDB environment.


4. Ensure your target NocoDB table schema matches the fields being inserted by the Set n8n node configurations (e.g., source_category, title, publisher, summary).

4. Enable Scheduling

Markdown
This n8n workflow utilizes several key n8n node types to manage scheduling, API connectivity, and data transformation:

Schedule Trigger (NewsAPI, Mediastack, CurrentsAPI)
Function: Acts as the starting n8n trigger for each separate aggregation stream.
Key Configuration: Set to run daily or bi-daily (Mediastack) at specific times (e.g., 6:00 AM).

Set (e.g., Set Newsapi.org top headlines, Set Mediastack)
Function: The essential normalization step. It standardizes disparate fields (e.g., urlToImage, source.name) into a consistent output schema (images, publisher).
Key Configuration: Contains complex expressions referencing the previous n8n node output (e.g., ={{ $json.data.title }}).

HTTP Request (Multiple instances)
Function: Connects to the external news APIs (NewsAPI, Mediastack, CurrentsAPI) to fetch raw article data.
Key Configuration: Uses query parameters or JSON bodies to pass required API keys, categories, and article limits. Configured with error handling (onError: continueErrorOutput) and retries for robustness.

Code (Itemize Categories)
Function: Converts the array of predefined categories from a Set n8n node into individual items, allowing subsequent nodes to iterate over them (e.g., fetch articles for 'technology', then 'sports').

Split Out / Split In Batches
Function: Manages data flow structure. Split Out unpacks the main article array from the API response. Split In Batches enables the workflow to loop efficiently over the categorized items.

NocoDB (Add item nodes)
Function: The final action n8n node, responsible for writing the clean, normalized data as new records into the centralized NocoDB database table.
Key Configuration: Uses the create operation and dynamically pulls data from the upstream Set n8n node outputs.