Twitter Profile Data Scraper using Bright Data and Google Sheets - n8n Workflow

Market Researchers: Need to track competitor or industry leader social activity over time.
Data Analysts: Requiring structured, historical social media data for analysis.
Growth Hackers: Automating lead generation or influence mapping by scraping key profile metrics.
n8n Users: Looking for advanced examples of how to implement API polling and looping within an n8n workflow.

Manually monitoring and collecting data from social media profiles is slow and prone to error, especially when dealing with large datasets or requiring specific date ranges. This advanced n8n workflow solves this by fully automating the process using a professional scraping provider, Bright Data.

This n8n workflow template is initiated by a simple user input (the n8n trigger), which requires only the target Twitter URL and a desired date range. The n8n logic then handles the complexity of asynchronous scraping tasks, including polling the Bright Data API every minute using the Wait n8n node until the data is marked as 'ready'. Once complete, the entire dataset is fetched and structured before being automatically appended to your designated Google Sheet using the Google Sheets n8n node. This specific n8n workflow ensures reliable, structured data extraction without needing complex scripting.

This comprehensive n8n workflow operates in a precise loop to manage the asynchronous nature of web scraping jobs:

1. User Input n8n Trigger: The workflow starts with an n8n Form Trigger that prompts the user for the target Twitter profile URL, a start date, and an end date for the scrape.


2. Trigger Scraping: An HTTP Request n8n node sends this data to the Bright Data API to initiate a new scraping job for the specified dataset.


3. Monitor Progress: A subsequent HTTP Request n8n node immediately polls the Bright Data API for the job's status using the unique snapshot_id.


4. Check and Delay Loop: The workflow enters a loop managed by two critical n8n nodes: the Wait node and the If node. The If n8n node checks if the status equals 'ready'. If the status is not ready, the workflow uses the Wait n8n node to pause for 1 minute before looping back to the Monitor Progress step, effectively polling the API until the job is done.


5. Fetch Data: Once the If n8n node confirms the status is 'ready', another HTTP Request n8n node downloads the full JSON data snapshot from Bright Data.


6. Store Data: Finally, the Google Sheets n8n node maps the collected attributes (including likes, views, followers, post content, and more) and appends them as new rows into the designated Google Sheet. This makes the data immediately available for analysis.

To deploy this n8n workflow template successfully, follow these steps:

1. Import the n8n Workflow: Copy the provided JSON and import it directly into your n8n instance.


2. Bright Data API Key: Open the '🚀 Trigger Twitter Scraping' and '🔄 Monitor Scraping Progress' n8n nodes. Replace the placeholder value BRIGHTDATAAPI_KEY in the Authorization header with your actual Bright Data API Bearer token.


3. Google Sheets Setup (Preparation): Create a new Google Sheet. According to the configuration in the final n8n node, ensure the first row contains column headers matching the expected output fields, such as id, name, likes, views, followers, timestamp, etc.


4. Google Sheets Credentials: Configure a Google Sheets OAuth2 credential in n8n. Then, open the '📊 Store Twitter Data in Google Sheet' n8n node and connect this credential.


5. Sheet Configuration: In the Google Sheets n8n node, update the Document ID and Sheet Name to point to your newly created spreadsheet. This final n8n node is crucial for storing the scraped data.


6. Activate: Save and activate the n8n workflow. Use the unique webhook ID provided by the '📥 User Input Trigger' n8n trigger to run your first test.

📥 User Input Trigger (Form Trigger): This serves as the manual n8n trigger for the entire process, capturing the profile URL and date range from the user.
🚀 Trigger Twitter Scraping (HTTP Request n8n node): Responsible for initiating the scraping job by POSTing the input data to the Bright Data API. Configuration includes passing inputs via JSON body and authenticating using the Authorization header.
🔄 Monitor Scraping Progress (HTTP Request n8n node): Polls the Bright Data API status endpoint, dynamically using the snapshotid returned from the trigger node via an expression.
⏱️ Delay Before Recheck (Wait n8n node): Essential for asynchronous polling. It pauses the n8n workflow execution for 1 minute before re-checking the status, preventing unnecessary load on the API.
✅ Is Scraping Ready? (If n8n node): Controls the loop structure. It checks if the status field ({{ $json.status }}) equals 'ready'. If True, the workflow progresses; if False, it cycles back through the Wait n8n node and Monitor node.
📦 Fetch Twitter Data (HTTP Request n8n node): Executed only when the scrape is ready. It downloads the complete, finalized data set from Bright Data.

• 📊 Store Twitter Data in Google Sheet (Google Sheets n8n node): The final action n8n node. It maps over twenty fields of rich Twitter data (e.g., likes, followers, postscount, is_verified) directly into the pre-configured Google Sheet, using the 'Append' operation.