Firecrawl Website Content Extraction Polling Loop - n8n Workflow

Data Analysts & Researchers: Those needing large volumes of structured web data extracted reliably.
SEO Specialists: Users who need to scrape competitor or industry content for analysis.
Developers: Individuals looking for reliable n8n templates to handle asynchronous API calls and complex retry logic.
n8n Users: Anyone building complex integrations who needs an example of a robust polling mechanism within an n8n workflow.

Handling long-running tasks, such as website scraping, requires managing asynchronous operations. When an initial request is sent, the extraction service often responds instantly with a Job ID, requiring the user to poll a status endpoint periodically until the job finishes. This n8n workflow template expertly solves this challenge. It initiates the Firecrawl extraction process, uses n8n node wait mechanisms to pause, and implements an If condition to continuously check the job status.

This design provides extreme reliability, ensuring the n8n workflow does not consume excessive resources while waiting, only proceeding to finalize the data once the extraction is 100% complete. This is a critical example of advanced flow control possible with n8n.

This comprehensive n8n workflow operates using a manual start and a tight polling loop to handle the asynchronous extraction job:

1. Trigger: The n8n workflow starts manually using the 'When clicking ‘Test workflow’' n8n trigger.


2. Initiate Extraction: The Extract n8n node (an HTTP Request) sends the initial API call to Firecrawl, starting the web extraction job and retrieving the unique Job ID.


3. Initial Wait: The workflow pauses for 30 seconds using the 30 Secs wait n8n node. This gives the extraction service time to complete smaller jobs.


4. Poll Status: The Get Results n8n node (another HTTP Request) uses the Job ID to query the status endpoint of the extraction API.


5. Conditional Check: The crucial If n8n node checks the API response data. If the extraction status is determined to be 'complete' (or similar success indicator), the workflow proceeds down the 'True' branch.


6. Finalization (True Branch): If complete, data flows to the Edit Fields n8n node for final restructuring and cleanup.


7. Polling Loop (False Branch): If the status is still 'processing' or 'pending', the data flows to the 'False' branch, hitting the 10 Seconds wait n8n node. After this shorter wait, the flow loops back to the Get Results n8n node to poll the status again. This creates an efficient, self-correcting polling mechanism within the n8n workflow.

To use this n8n template, follow these steps:

1. Import: Copy the entire n8n workflow JSON provided and paste it into your n8n instance using the 'New' -> 'Import from JSON' feature.


2. API Credentials: The HTTP Request nodes are configured to hit a generic endpoint. You must update both the Extract and Get Results n8n node configurations to use your specific Firecrawl or extraction service credentials (API Key via header or basic auth) and correct URL endpoints.


3. Payload Configuration: In the Extract n8n node, configure the request body to specify the URL you wish to scrape.


4. Wait Times: Adjust the 30 Secs and 10 Seconds n8n node wait times based on the average duration of your extraction jobs.


5. Test: Click the 'Test workflow' button on the 'When clicking ‘Test workflow’' n8n trigger to ensure the entire polling loop functions correctly.

When clicking ‘Test workflow’ (Manual Trigger): The starting point and primary n8n trigger for running this n8n workflow manually for testing or development.
Extract (HTTP Request n8n node): Sends the initial request to the Firecrawl API to start the scraping job. It captures the resulting Job ID required for subsequent status checks.
30 Secs (Wait n8n node): Introduces an initial, longer delay to prevent immediate excessive polling, optimizing resource use within the n8n workflow.
Get Results (HTTP Request n8n node): Used within the polling loop. This n8n node queries the API status endpoint using the Job ID to determine if the extraction task is finished.
If (If n8n node): The core control mechanism of the n8n workflow. It evaluates the status returned by the Get Results node. If the job status meets the completion criteria, it exits the loop; otherwise, it sends the process back to wait.
10 Seconds (Wait n8n node): A shorter wait period used when the job is still processing. It controls the frequency of polling checks, preventing API rate limit issues while running the n8n workflow.

• Edit Fields (Set n8n node): The final processing step on the 'True' path. It is used to clean, rename, or structure the extracted content before sending it to the next step in a larger n8n workflow.