Asynchronous Bulk Web Scraping with Bright Data & Webhook Polling - n8n Workflow

-
Data Analysts and Scientists needing to collect large, structured datasets from the web.
Developers building data pipelines who require reliable, asynchronous scraping methods.
n8n users looking for advanced n8n templates demonstrating polling loops and external API integration.
Engineering teams seeking to monitor and process bulk data extraction jobs efficiently.

Handling bulk web scraping jobs often requires asynchronous processing: you initiate the job, and then you must periodically check if the results are ready. This powerful n8n workflow solves this complexity by leveraging Bright Data's infrastructure alongside core n8n logic flow control. The initial n8n trigger sets up the scraping request. The subsequent steps create a polling mechanism, ensuring the workflow waits patiently for the snapshot to be ready without timing out. Once the data is confirmed ready and error-free, the n8n workflow downloads the structured JSON, aggregates it, sends a notification webhook, and optionally writes the final data to a local disk. This sophisticated n8n template is ideal for scalable data acquisition.

The entire process begins with a manual n8n trigger (When clicking ‘Test workflow’).

1. Setup and Initiation: The first n8n node, 'Set Dataset Id, Request URL', defines the target Bright Data Dataset ID and the specific URL(s) to scrape (currently configured for an Amazon product page). The subsequent 'HTTP Request to the specified URL' n8n node uses a POST request to Bright Data's datasets/v3/trigger endpoint to start the scraping job.


2. Snapshot ID Capture: The next n8n node, 'Set Snapshot Id', captures the unique identifier needed to track the job's progress.


3. Polling Loop: The 'Check Snapshot Status' n8n node sends a GET request to Bright Data to query the status of the job using the snapshot ID. The 'If' n8n node then evaluates the response. If the status is not 'ready', the workflow executes the 'Wait' n8n node for 30 seconds before looping back to 'Check Snapshot Status'. This polling loop continues until the job is complete.


4. Error and Download: Once the status is 'ready', the 'Check on the errors' n8n node ensures no critical failures occurred. If clean, the 'Download Snapshot' n8n node retrieves the collected data.


5. Final Processing and Output: The 'Aggregate JSON Response' n8n node combines the downloaded segments. Finally, the n8n workflow branches: it sends a notification payload to an external URL ('Initiate a Webhook Notification') and simultaneously uses the 'Create a binary data' Function n8n node and the 'Write the file to disk' n8n node to save the resulting JSON data locally.

To deploy and utilize this n8n workflow effectively, follow these steps:

1. Import: Copy the provided JSON code and import it directly into your n8n instance using the 'New' menu and selecting 'Import workflow from JSON'.


2. Credentials: This n8n workflow requires a Bright Data authentication setup. Configure an 'HTTP Header Auth' credential using your Bright Data API token (often passed as the Authorization header or similar custom header setup defined by Bright Data's API documentation).


3. Configuration Nodes:

1. Test: Run the n8n trigger manually to test the full asynchronous data flow.

When clicking ‘Test workflow’ (Manual Trigger): This is the essential starting n8n trigger. It allows the user to initiate the bulk scraping job manually.
Set Dataset Id, Request URL (Set n8n node): Defines necessary variables like the Bright Data datasetid and the JSON array containing the target request URLs for scraping.
HTTP Request to the specified URL (HTTP Request n8n node): Initiates the Bright Data job via the /datasets/v3/trigger API endpoint. Key configuration includes using the POST method and passing the request JSON in the body.
Check Snapshot Status (HTTP Request n8n node): Used within the polling loop. It checks the progress of the scraping job using the /datasets/v3/progress/ endpoint, leveraging the snapshotid saved earlier.
If (If n8n node): Crucial for flow control. Determines if the job status received from Bright Data is 'ready'. This is how the n8n workflow decides to continue or wait.
Wait (Wait n8n node): If the job is not ready, this n8n node pauses the workflow for 30 seconds before re-checking the status, enabling asynchronous handling.
Download Snapshot (HTTP Request n8n node): Once ready, this n8n node fetches the final structured data from the /datasets/v3/snapshot/ endpoint, configured to retrieve data in JSON format.
Aggregate JSON Response (Aggregate n8n node): Ensures that all data items downloaded are combined into a single, cohesive payload for subsequent processing.
Initiate a Webhook Notification (HTTP Request n8n node): Sends the final processed data or a summary notification to an external service, confirming the successful completion of this n8n workflow run.
Write the file to disk (Read/Write File n8n node): Writes the final structured JSON data to a specified local path, providing a permanent record of the scrape.